This directory contains an SGLang component for Dynamo and reference implementations for deploying Large Language Models (LLMs) in various configurations using SGLang. SGLang internally uses ZMQ to communicate between the ingress and the engine processes. For Dynamo, we leverage the runtime to communicate directly with the engine processes and handle ingress and pre/post processing on our end.
## Use the Latest Release
We recommend using the latest stable release of dynamo to avoid breaking changes:
| `--endpoint` | Dynamo endpoint in `dyn://namespace.component.endpoint` format | Auto-generated based on mode | N/A |
| `--migration-limit` | Max times a request can migrate between workers | `0` (disabled) | N/A |
| `--migration-limit` | Max times a request can migrate between workers for fault tolerance. See [Request Migration Architecture](../../../docs/architecture/request_migration.md). | `0` (disabled) | N/A |
| `--dyn-tool-call-parser` | Tool call parser for structured outputs (takes precedence over `--tool-call-parser`) | `None` | `--tool-call-parser` |
| `--dyn-reasoning-parser` | Reasoning parser for CoT models (takes precedence over `--reasoning-parser`) | `None` | `--reasoning-parser` |
| `--use-sglang-tokenizer` | Use SGLang's tokenizer instead of Dynamo's | `False` | N/A |
...
...
@@ -73,39 +63,33 @@ Dynamo SGLang uses SGLang's native argument parser, so **most SGLang engine argu
#### Tokenizer Behavior
-**Default (`--use-sglang-tokenizer` not set)**: Dynamo handles tokenization and passes `input_ids` to SGLang
-**With `--use-sglang-tokenizer`**: SGLang handles tokenization, Dynamo passes raw prompts
> **Note**: When using `--use-sglang-tokenizer`, only `v1/chat/completions` endpoints are available through Dynamo's frontend.
## SGLang Quick Start
Below we provide a guide that lets you run all of our common deployment patterns on a single node.
### Start NATS and ETCD in the background
Start using [Docker Compose](../../../deploy/docker-compose.yml)
-**Default (`--use-sglang-tokenizer` not set)**: Dynamo handles tokenization/detokenization via our blazing fast frontend and passes `input_ids` to SGLang
-**With `--use-sglang-tokenizer`**: SGLang handles tokenization/detokenization, Dynamo passes raw prompts
```bash
docker compose -f deploy/docker-compose.yml up -d
```
> [!NOTE]
> When using `--use-sglang-tokenizer`, only `v1/chat/completions` is available through Dynamo's frontend.
### Install `ai-dynamo[sglang]`
## Installation
#### Install latest release
### Install latest release
We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with `curl -LsSf https://astral.sh/uv/install.sh | sh`
<details>
<summary>Expand for instructions</summary>
```bash
# create a virtual env
uv venv --python 3.12 --seed
# install the latest release
# install the latest release (which comes bundled with a stable sglang version)
uv pip install"ai-dynamo[sglang]"
```
#### Installing editable version for development
</details>
### Install editable version for development
<details>
<summary>Instructions</summary>
<summary>Expand for instructions</summary>
This requires having rust installed. We also recommend having a proper installation of the cuda toolkit as sglang requires `nvcc` to be available.
...
...
@@ -119,40 +103,61 @@ maturin develop --uv
cd$DYNAMO_HOME
# installs sglang supported version along with dynamo
# include the prerelease flag to install flashinfer rc versions
uv pip install--prerelease=allow -e .[sglang]
uv pip install-e.
# install any sglang version >= 0.5.3
uv pip install"sglang[all]==0.5.3.post1"
```
</details>
#### Using prebuilt docker containers
### Using docker containers
<details>
<summary>Instructions</summary>
<summary>Expand for instructions</summary>
We are in the process of shipping pre-built docker containers that contain installations of DeepEP, DeepGEMM, and NVSHMEM in order to support WideEP and P/D. For now, you can quickly build the container from source with the following command.
Below we provide a guide that lets you run all of our common deployment patterns on a single node.
### Start NATS and ETCD in the background
> [!IMPORTANT]
Start using [Docker Compose](../../../deploy/docker-compose.yml)
```bash
docker compose -f deploy/docker-compose.yml up -d
```
> [!TIP]
> Each example corresponds to a simple bash script that runs the OpenAI compatible server, processor, and optional router (written in Rust) and LLM engine (written in Python) in a single terminal. You can easily take each command and run them in separate terminals.
>
> Additionally - because we use sglang's argument parser, you can pass in any argument that sglang supports to the worker!
...
...
@@ -167,15 +172,12 @@ cd $DYNAMO_HOME/components/backends/sglang
### Aggregated Serving with KV Routing
> [!NOTE]
> Until sglang releases a version > v0.5.0rc0, you will have to install from source to use kv_routing. You can do this by running `git clone https://github.com/sgl-project/sglang.git && cd sglang && uv pip install -e "python[all]"`. We will update this section once sglang releases a newer version.
```bash
cd$DYNAMO_HOME/components/backends/sglang
./launch/agg_router.sh
```
### Aggregated Serving with Embeddings
### Aggregated Serving for Embedding Models
Here's an example that uses the [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) model.
...
...
@@ -184,7 +186,8 @@ cd $DYNAMO_HOME/components/backends/sglang
./launch/agg_embed.sh
```
Send the following request to verify your deployment:
<details>
<summary>Send the following request to verify your deployment:</summary>
<summary>Under the hood: SGLang Load Balancer vs Dynamo Discovery</summary>
</details>
SGLang uses a mini load balancer to route requests to handle disaggregated serving. The load balancer functions as follows:
### Disaggregated serving
1. The load balancer receives a request from the client
2. A random `(prefill, decode)` pair is selected from the pool of available workers
3. Request is sent to both `prefill` and `decode` workers via asyncio tasks
4. Internally disaggregation is done from prefill -> decode
See [SGLang Disaggregation](sglang-disaggregation.md) to learn more about how sglang and dynamo handle disaggregated serving.
Because Dynamo has a discovery mechanism, we do not use a load balancer. Instead, we first route to a random prefill worker, select a random decode worker, and then send the request to both. Internally, SGLang's bootstrap server (which is a part of the `tokenizer_manager`) is used in conjuction with NIXL to handle the kv transfer.
</details>
```bash
cd$DYNAMO_HOME/components/backends/sglang
./launch/disagg.sh
```
> [!IMPORTANT]
> Disaggregated serving in SGLang currently requires each worker to have the same tensor parallel size [unless you are using an MLA based model](https://github.com/sgl-project/sglang/pull/5922)
### Disaggregated Serving with KV Aware Prefill Routing
```bash
cd$DYNAMO_HOME/components/backends/sglang
./launch/disagg.sh
./launch/disagg_router.sh
```
### Disaggregated Serving with Mixture-of-Experts (MoE) models and DP attention
...
...
@@ -229,8 +227,6 @@ cd $DYNAMO_HOME/components/backends/sglang
./launch/disagg_dp_attn.sh
```
When using MoE models, you can also use the our implementation of the native SGLang endpoints to record expert distribution data. The `disagg_dp_attn.sh` script automatically sets up the SGLang HTTP server, the environment variable that controls the expert distribution recording directory, and sets up the expert distribution recording mode to `stat`. You can learn more about expert parallelism load balancing [here](expert-distribution-eplb.md).
You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
```bash
python3 -m dynamo.sglang ... --migration-limit=3
```
This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.
## Advanced Examples
Below we provide a selected list of advanced examples. Please open up an issue if you'd like to see a specific example!
...
...
@@ -269,7 +255,7 @@ Below we provide a selected list of advanced examples. Please open up an issue i
-**[Run a multi-node model](multinode-examples.md)**
### Large scale P/D disaggregation with WideEP
-**[Run DeepSeek-R1 on 104+ H100s](dsr1-wideep-h100.md)**
-**[Run DeepSeek-R1-FP8 on H100s](dsr1-wideep-h100.md)**
-**[Run DeepSeek-R1-FP8 on GB200s](dsr1-wideep-gb200.md)**
@@ -17,21 +17,21 @@ limitations under the License.
# Running DeepSeek-R1 Disaggregated with WideEP on GB200s
Dynamo supports SGLang's GB200 implementation of wide expert parallelism and large scale P/D for DeepSeek-R1! You can read their blog post [here](https://lmsys.org/blog/2025-06-16-gb200-part-1/) for more details. Full end to end optimization is still a work in progress but you can get this up and running with the following steps. In ths example, we will run 1 prefill worker on 2 GB200 nodes (4 GPUs each) and 1 decode worker on 12 GB200 nodes (total 56 GPUs).
Dynamo supports SGLang's GB200 implementation of wide expert parallelism and large scale P/D for DeepSeek-R1! You can read their blog post [here](https://lmsys.org/blog/2025-06-16-gb200-part-1/) for more details. We provide a Dockerfile for this in `container/Dockerfile.sglang-wideep` and a sample configuration that demonstrates WideEP and P/D disaggregation. To run the exact configuration shown in the blog post, you can view the commands created by the SGLang team [here](https://github.com/sgl-project/sglang/issues/7227). In this example, we will run 1 prefill worker on 2 GB200 nodes (4 GPUs each) and 1 decode worker on 2 GB200 nodes (total 8 GPUs).
## Instructions
1. Build the Dynamo container
1. Build the Dynamo container using the latest published dynamo version and stable sglang version. If you want to build from a local dynamo repo, you can add `--build-arg BRANCH_TYPE=local` to the build command. If you want to build from a remote dynamo repo, you can add `--build-arg BRANCH_TYPE=remote` to the build command. If you want to use a specific tag for the default sglang version, you can add `--build-arg SGLANG_IMAGE_TAG=<tag>` to the build command.
> [!Note]
> Please ensure that you are building this on an ARM64 machine. The correct SGLang image will be selected automatically via the multi-arch manifest.
# Running DeepSeek-R1 Disaggregated with WideEP on H100s
Dynamo supports SGLang's implementation of wide expert parallelism and large scale P/D for DeepSeek-R1! You can read their blog post [here](https://lmsys.org/blog/2025-05-05-large-scale-ep/) for more details. We provide a Dockerfile for this in `container/Dockerfile.sglang-wideep` and configurations to deploy this at scale. In this example, we will run 1 prefill worker on 4 H100 nodes and 1 decode worker on 9 H100 nodes (104 total GPUs).
Dynamo supports SGLang's implementation of wide expert parallelism and large scale P/D for DeepSeek-R1! You can read their blog post [here](https://lmsys.org/blog/2025-05-05-large-scale-ep/) for more details. We provide a Dockerfile for this in `container/Dockerfile.sglang-wideep` and a sample configuration that demonstrates WideEP and P/D disaggregation. To run the exact configuration shown in the blog post, you can view the commands created by the SGLang team [here](https://github.com/sgl-project/sglang/issues/6017). In this example, we will run 1 prefill worker on 4 H100 nodes and 1 decode worker on 4 H100 nodes (64 total GPUs).
## Instructions
1. Build the Dynamo container
1. Build the Dynamo container using the latest published dynamo version and stable sglang version. If you want to build from a local dynamo repo, you can add `--build-arg BRANCH_TYPE=local` to the build command. If you want to build from a remote dynamo repo, you can add `--build-arg BRANCH_TYPE=remote` to the build command. If you want to use a specific tag for the default sglang version, you can add `--build-arg SGLANG_IMAGE_TAG=<tag>` to the build command.
> [!Note]
> Please ensure that you are building this on an AMD64 (x86_64) machine. The correct SGLang image will be selected automatically via the multi-arch manifest.
You can use a specific tag from the [lmsys dockerhub](https://hub.docker.com/r/lmsysorg/sglang/tags) by adding `--build-arg SGLANG_IMAGE_TAG=<tag>` to the build command.
2. You can run this container on each 8xH100 node using the following command.
> [!IMPORTANT]
...
...
@@ -46,8 +51,6 @@ In each container, you should be in the `/sgl-workspace/dynamo/components/backen
```bash
# run ingress
python3 -m dynamo.frontend --http-port=8000 &
# optionally run the http server that allows you to flush the kv cache for all workers (see benchmarking section below)
python3 utils/sgl_http_server.py --ns dynamo &
# run prefill worker
python3 -m dynamo.sglang \
--model-path /model/ \
...
...
@@ -62,8 +65,9 @@ python3 -m dynamo.sglang \
--tp-size 32 \
--dp-size 32 \
--enable-dp-attention\
--decode-log-interval 1 \
--enable-deepep-moe\
--decode-log-interval 1000 \
--moe-a2a-backend deepep \
--load-balance-method round_robin \
--page-size 1 \
--trust-remote-code\
--moe-dense-tp-size 1 \
...
...
@@ -92,13 +96,14 @@ python3 -m dynamo.sglang \
--disaggregation-transfer-backend nixl \
--disaggregation-bootstrap-port 30001 \
--dist-init-addr${HEAD_DECODE_NODE_IP}:29500 \
--nnodes9\
--nnodes4\
--node-rank 0 \
--tp-size72 \
--dp-size72 \
--tp-size32 \
--dp-size32 \
--enable-dp-attention\
--decode-log-interval 1 \
--enable-deepep-moe\
--decode-log-interval 1000 \
--moe-a2a-backend deepep \
--prefill-round-robin-balance\
--page-size 1 \
--trust-remote-code\
--moe-dense-tp-size 1 \
...
...
@@ -112,59 +117,4 @@ python3 -m dynamo.sglang \
--cuda-graph-bs 128
```
On the other decode nodes (this example has 9 total decode nodes), run the same command but change `--node-rank` to 1, 2, 3, 4, 5, 6, 7, and 8
## Benchmarking
In the official [blog post repro instructions](https://github.com/sgl-project/sglang/issues/6017), SGL uses batch inference to benchmark their prefill and decode workers. They do this by pretokenizing the ShareGPT dataset and then creating a batch of 8192 requests with ISL 4096 and OSL 5 (for prefill stress test) and a batch of 40000 with ISL 2000 and OSL 100 (for decode stress test). If you want to repro these benchmarks, you will need to add the following flags to the prefill and decode commands:
We currently provide 2 different ways to perform an end to end benchmark which includes using our OpenAI frontend and tokenization. We will continue to add better support for these sorts of large single batch workloads in the future.
1.**GenAI Perf to benchmark end to end performance with 8k ISL 256 OSL**
We've found that 8k ISL 256 OSL provides a good baseline for measuring end to end disaggregated serving performance for DSR1. As WideEP allows for a higher throughput, we provide a script that runs this workload at high concurrencies. DeepGEMM kernels can sometimes take a while to warm up. We provide a short ramping warmup script that can be used.
# if you ran the http server on the head prefill node, you can optionally flush the kv cache for all workers (similar to SGLangs benchmarking script)
curl -X POST http://${HEAD_PREFILL_NODE_IP}:9001/flush_cache
# run benchmark
./utils/bench.sh HEAD_PREFILL_NODE_IP --type e2e
```
2.**GenAI Perf to benchmark completions with custom dataset**
We provide a script that generates a JSONL file of the ShareGPT dataset and then use GenAI Perf to benchmark the prefill and decode workers. We use ShareGPT in order to leverage the pre-existing EPLB distributions provided by the SGLang team. If you don't want to use ShareGPT - you can also use GenAI Perf's synthetic dataset setup But note you will have to use dynamic EPLB configurations or record your own as the `init-expert-location` provided by SGLang is tuned specifically for the ShareGPT dataset at a 4096 ISL and 5 OSL.
