Unverified Commit 269c457e authored by Michael Yao's avatar Michael Yao Committed by GitHub
Browse files

[Docs] Update runtime/engine/readme.md (#5737)


Signed-off-by: default avatarwindsonsea <haifeng.yao@daocloud.io>
parent 18ce468d
# SGLang Engine # SGLang Engine
## Introduction SGLang provides a direct inference engine without the need for an HTTP server. There are generally these use cases:
SGLang provides a direct inference engine without the need for an HTTP server. There are generally two use cases:
1. **Offline Batch Inference** - [Offline Batch Inference](#offline-batch-inference)
2. **Embedding Generation** - [Embedding Generation](#embedding-generation)
3. **Custom Server on Top of the Engine** - [Custom Server](#custom-server)
4. **Inference Using FastAPI** - [Token-In-Token-Out for RLHF](#token-in-token-out-for-rlhf)
- [Inference Using FastAPI](#inference-using-fastapi)
## Examples ## Examples
...@@ -22,28 +22,28 @@ In this example, we launch an SGLang engine and feed a batch of inputs for embed ...@@ -22,28 +22,28 @@ In this example, we launch an SGLang engine and feed a batch of inputs for embed
This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints. This example demonstrates how to create a custom server on top of the SGLang Engine. We use [Sanic](https://sanic.dev/en/) as an example. The server supports both non-streaming and streaming endpoints.
#### Steps: #### Steps
1. Install Sanic: 1. Install Sanic:
```bash ```bash
pip install sanic pip install sanic
``` ```
2. Run the server: 2. Run the server:
```bash ```bash
python custom_server python custom_server
``` ```
3. Send requests: 3. Send requests:
```bash ```bash
curl -X POST http://localhost:8000/generate -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}' curl -X POST http://localhost:8000/generate -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}'
curl -X POST http://localhost:8000/generate_stream -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}' --no-buffer curl -X POST http://localhost:8000/generate_stream -H "Content-Type: application/json" -d '{"prompt": "The Transformer architecture is..."}' --no-buffer
``` ```
This will send both non-streaming and streaming requests to the server. This will send both non-streaming and streaming requests to the server.
### [Token-In-Token-Out for RLHF](../token_in_token_out) ### [Token-In-Token-Out for RLHF](../token_in_token_out)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment