-[Token-In-Token-Out for RLHF](#token-in-token-out-for-rlhf)
-[Inference Using FastAPI](#inference-using-fastapi)
## Examples
## Examples
...
@@ -22,28 +22,28 @@ In this example, we launch an SGLang engine and feed a batch of inputs for embed
...
@@ -22,28 +22,28 @@ In this example, we launch an SGLang engine and feed a batch of inputs for embed
This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints.
This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints.
#### Steps:
#### Steps
1. Install Sanic:
1. Install Sanic:
```bash
```bash
pip install sanic
pip install sanic
```
```
2. Run the server:
2. Run the server:
```bash
```bash
python custom_server
python custom_server
```
```
3. Send requests:
3. Send requests:
```bash
```bash
curl -X POST http://localhost:8000/generate -H"Content-Type: application/json"-d'{"prompt": "The Transformer architecture is..."}'
curl -X POST http://localhost:8000/generate -H"Content-Type: application/json"-d'{"prompt": "The Transformer architecture is..."}'
curl -X POST http://localhost:8000/generate_stream -H"Content-Type: application/json"-d'{"prompt": "The Transformer architecture is..."}'--no-buffer
curl -X POST http://localhost:8000/generate_stream -H"Content-Type: application/json"-d'{"prompt": "The Transformer architecture is..."}'--no-buffer
```
```
This will send both non-streaming and streaming requests to the server.
This will send both non-streaming and streaming requests to the server.
### [Token-In-Token-Out for RLHF](../token_in_token_out)
### [Token-In-Token-Out for RLHF](../token_in_token_out)