README.md 10.6 KB
Newer Older
1
2
3
4
5
<!--
SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

6
# Running SGLang with Dynamo
7

8
This directory contains an SGLang component for Dynamo and reference implementations for deploying Large Language Models (LLMs) in various configurations using SGLang. SGLang internally uses ZMQ to communicate between the ingress and the engine processes. For Dynamo, we leverage the runtime to communicate directly with the engine processes and handle ingress and pre/post processing on our end.
9

10
11
12
13
14
15
16
17
18
19
20
21
## Use the Latest Release

We recommend using the latest stable release of dynamo to avoid breaking changes:

[![GitHub Release](https://img.shields.io/github/v/release/ai-dynamo/dynamo)](https://github.com/ai-dynamo/dynamo/releases/latest)

You can find the latest release [here](https://github.com/ai-dynamo/dynamo/releases/latest) and check out the corresponding branch with:

```bash
git checkout $(git describe --tags $(git rev-list --tags --max-count=1))
```

22
---
23

24
25
## Table of Contents
- [Feature Support Matrix](#feature-support-matrix)
26
- [Dynamo SGLang Integration](#dynamo-sglang-integration)
27
28
29
30
- [Quick Start](#quick-start)
- [Single Node Examples](#run-single-node-examples)
- [Multi-Node and Advanced Examples](#advanced-examples)
- [Deploy on SLURM or Kubernetes](#deployment)
31

32
## Feature Support Matrix
33

34
### Core Dynamo Features
35

36
37
| Feature | SGLang | Notes |
|---------|--------|-------|
38
39
40
| [**Disaggregated Serving**](../../../docs/architecture/disagg_serving.md) | ✅ |  |
| [**Conditional Disaggregation**](../../../docs/architecture/disagg_serving.md#conditional-disaggregation) | 🚧 | WIP [PR](https://github.com/sgl-project/sglang/pull/7730) |
| [**KV-Aware Routing**](../../../docs/architecture/kv_cache_routing.md) | ✅ |  |
41
| [**SLA-Based Planner**](../../../docs/architecture/sla_planner.md) | ✅ |  |
42
43
| [**Load Based Planner**](../../../docs/architecture/load_planner.md) | ❌ | Planned |
| [**KVBM**](../../../docs/architecture/kvbm_architecture.md) | ❌ | Planned |
44

45
46
### Large Scale P/D and WideEP Features

ishandhanani's avatar
ishandhanani committed
47
48
49
50
51
| Feature             | SGLang | Notes                                                        |
|---------------------|--------|--------------------------------------------------------------|
| **WideEP**          | ✅     | Full support on H100s/GB200                                  |
| **DP Rank Routing** | 🚧     | Direct routing supported. Dynamo KV router does not router to DP worker |
| **GB200 Support**   | ✅     |                                                              |
52
53


54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
## Dynamo SGLang Integration

Dynamo SGLang integrates SGLang engines into Dynamo's distributed runtime, enabling advanced features like disaggregated serving, KV-aware routing, and request migration while maintaining full compatibility with SGLang's engine arguments.

### Argument Handling

Dynamo SGLang uses SGLang's native argument parser, so **most SGLang engine arguments work identically**. You can pass any SGLang argument (like `--model-path`, `--tp`, `--trust-remote-code`) directly to `dynamo.sglang`.

#### Dynamo-Specific Arguments

| Argument | Description | Default | SGLang Equivalent |
|----------|-------------|---------|-------------------|
| `--endpoint` | Dynamo endpoint in `dyn://namespace.component.endpoint` format | Auto-generated based on mode | N/A |
| `--migration-limit` | Max times a request can migrate between workers | `0` (disabled) | N/A |
| `--dyn-tool-call-parser` | Tool call parser for structured outputs (takes precedence over `--tool-call-parser`) | `None` | `--tool-call-parser` |
| `--dyn-reasoning-parser` | Reasoning parser for CoT models (takes precedence over `--reasoning-parser`) | `None` | `--reasoning-parser` |
| `--use-sglang-tokenizer` | Use SGLang's tokenizer instead of Dynamo's | `False` | N/A |
71
| `--custom-jinja-template` | Use custom chat template for that model (takes precedence over default chat template in model repo) | `None` | `--chat-template` |
72
73
74
75
76
77
78
79

#### Tokenizer Behavior

- **Default (`--use-sglang-tokenizer` not set)**: Dynamo handles tokenization and passes `input_ids` to SGLang
- **With `--use-sglang-tokenizer`**: SGLang handles tokenization, Dynamo passes raw prompts

> **Note**: When using `--use-sglang-tokenizer`, only `v1/chat/completions` endpoints are available through Dynamo's frontend.

80
## SGLang Quick Start
81

82
Below we provide a guide that lets you run all of our common deployment patterns on a single node.
83
84
85

### Start NATS and ETCD in the background

86
Start using [Docker Compose](../../../deploy/docker-compose.yml)
87

88
```bash
89
docker compose -f deploy/docker-compose.yml up -d
90
91
```

92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
### Install `ai-dynamo[sglang]`

#### Install latest release
We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with `curl -LsSf https://astral.sh/uv/install.sh | sh`

```bash
# create a virtual env
uv venv --python 3.12 --seed
# install the latest release
uv pip install "ai-dynamo[sglang]"
```

#### Installing editable version for development

<details>
<summary>Instructions</summary>

This requires having rust installed. We also recommend having a proper installation of the cuda toolkit as sglang requires `nvcc` to be available.

```bash
# create a virtual env
uv venv --python 3.12 --seed
# build dynamo runtime bindings
uv pip install maturin
cd $DYNAMO_HOME/lib/bindings/python
maturin develop --uv
cd $DYNAMO_HOME
uv pip install .
export PYTHONPATH="${PYTHONPATH}:$(pwd)/components/backends/sglang/src"
# install target sglang version (you can choose any version)
# we include the prerelease flag in order to install flashinfer rc versions
uv pip install --prerelease=allow sglang[all]==0.4.9.post6
```

</details>

#### Using prebuilt docker containers

<details>
<summary>Instructions</summary>
132
133

```bash
134
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:my-tag
135
136
```

137
138
139
140
141
142
</details>

#### Building docker container from source

<details>
<summary>Instructions</summary>
143
144

```bash
145
146
147
148
149
./container/build.sh --framework sglang
# run container using prebuild wheel
./container/run.sh --framework sglang -it
# mount workspace for development
./container/run.sh --framework sglang --mount-workspace
150
151
```

152
153
</details>

154
## Run Single Node Examples
155

156
> [!IMPORTANT]
157
158
159
> Each example corresponds to a simple bash script that runs the OpenAI compatible server, processor, and optional router (written in Rust) and LLM engine (written in Python) in a single terminal. You can easily take each command and run them in separate terminals.
>
> Additionally - because we use sglang's argument parser, you can pass in any argument that sglang supports to the worker!
160

161
162

### Aggregated Serving
163
164

```bash
165
cd $DYNAMO_HOME/components/backends/sglang
166
./launch/agg.sh
167
```
168

169
### Aggregated Serving with KV Routing
170

171
172
173
> [!NOTE]
> Until sglang releases a version > v0.5.0rc0, you will have to install from source to use kv_routing. You can do this by running `git clone https://github.com/sgl-project/sglang.git && cd sglang && uv pip install -e "python[all]"`. We will update this section once sglang releases a newer version.

174
```bash
175
cd $DYNAMO_HOME/components/backends/sglang
176
./launch/agg_router.sh
177
178
```

179
### Disaggregated serving
180

181
<details>
182
<summary>Under the hood: SGLang Load Balancer vs Dynamo Discovery</summary>
183
184

SGLang uses a mini load balancer to route requests to handle disaggregated serving. The load balancer functions as follows:
185
186
187
188
189
190
191
192

1. The load balancer receives a request from the client
2. A random `(prefill, decode)` pair is selected from the pool of available workers
3. Request is sent to both `prefill` and `decode` workers via asyncio tasks
4. Internally disaggregation is done from prefill -> decode

Because Dynamo has a discovery mechanism, we do not use a load balancer. Instead, we first route to a random prefill worker, select a random decode worker, and then send the request to both. Internally, SGLang's bootstrap server (which is a part of the `tokenizer_manager`) is used in conjuction with NIXL to handle the kv transfer.

193
194
</details>

195
196
197
198
> [!IMPORTANT]
> Disaggregated serving in SGLang currently requires each worker to have the same tensor parallel size [unless you are using an MLA based model](https://github.com/sgl-project/sglang/pull/5922)

```bash
199
cd $DYNAMO_HOME/components/backends/sglang
200
./launch/disagg.sh
201
```
202

203
### Disaggregated Serving with Mixture-of-Experts (MoE) models and DP attention
204

205
You can use this configuration to test out disaggregated serving with dp attention and expert parallelism on a single node before scaling to the full DeepSeek-R1 model across multiple nodes.
206
207
208

```bash
# note this will require 4 GPUs
209
cd $DYNAMO_HOME/components/backends/sglang
210
./launch/disagg_dp_attn.sh
211
```
212

213
214
When using MoE models, you can also use the our implementation of the native SGLang endpoints to record expert distribution data. The `disagg_dp_attn.sh` script automatically sets up the SGLang HTTP server, the environment variable that controls the expert distribution recording directory, and sets up the expert distribution recording mode to `stat`. You can learn more about expert parallelism load balancing [here](docs/expert-distribution-eplb.md).

215
216
217
218
219
220
221
222
### Testing the Deployment

Send a test request to verify your deployment:

```bash
curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
223
    "model": "Qwen/Qwen3-0.6B",
224
225
226
227
228
229
    "messages": [
    {
        "role": "user",
        "content": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
    }
    ],
230
    "stream": true,
231
232
233
234
    "max_tokens": 30
  }'
```

235
236
## Request Migration

237
You can enable [request migration](../../../docs/architecture/request_migration.md) to handle worker failures gracefully. Use the `--migration-limit` flag to specify how many times a request can be migrated to another worker:
238
239
240
241
242

```bash
python3 -m dynamo.sglang ... --migration-limit=3
```

243
This allows a request to be migrated up to 3 times before failing. See the [Request Migration Architecture](../../../docs/architecture/request_migration.md) documentation for details on how this works.
244

245
246
247
248
## Advanced Examples

Below we provide a selected list of advanced examples. Please open up an issue if you'd like to see a specific example!

ishandhanani's avatar
ishandhanani committed
249
### Run a multi-node sized model
250
251
252
253
- **[Run a multi-node model](docs/multinode-examples.md)**

### Large scale P/D disaggregation with WideEP
- **[Run DeepSeek-R1 on 104+ H100s](docs/dsr1-wideep-h100.md)**
254
- **[Run DeepSeek-R1-FP8 on GB200s](docs/dsr1-wideep-gb200.md)**
255

256
257
258
### Hierarchical Cache (HiCache)
- **[Enable SGLang Hierarchical Cache (HiCache)](docs/sgl-hicache-example.md)**

259
260
## Deployment

261
We currently provide deployment examples for Kubernetes and SLURM.
262

263
## Kubernetes
264
- **[Deploying Dynamo with SGLang on Kubernetes](deploy/README.md)**
265

266
## SLURM
267
- **[Deploying Dynamo with SGLang on SLURM](slurm_jobs/README.md)**