"vscode:/vscode.git/clone" did not exist on "6e8f7605187c2699a18e815c646a4f21e456defa"
Unverified Commit c3f2fc5a authored by Byron Hsu's avatar Byron Hsu Committed by GitHub
Browse files

[doc] Add engine section in backend.md (#1656)

parent 7ee6c259
...@@ -93,6 +93,15 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct ...@@ -93,6 +93,15 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1
``` ```
### SRT Engine: Direct Inference Without HTTP
SGLang provides a direct inference engine **without an HTTP server**. This can be used for:
1. **Offline Batch Inference**
2. **Building Custom Servers**
We provide usage examples [here](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine)
### Supported Models ### Supported Models
**Generative Models** **Generative Models**
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment