README.md 13.5 KB
Newer Older
Neelay Shah's avatar
Neelay Shah committed
1
<!--
Neelay Shah's avatar
Neelay Shah committed
2
SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Neelay Shah's avatar
Neelay Shah committed
3
SPDX-License-Identifier: Apache-2.0
4
5
6
7
8
9
10
11
12
13
14
15

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Neelay Shah's avatar
Neelay Shah committed
16
-->
17

18
![Dynamo banner](./docs/images/frontpage-banner.png)
Neelay Shah's avatar
Neelay Shah committed
19

20
21
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![GitHub Release](https://img.shields.io/github/v/release/ai-dynamo/dynamo)](https://github.com/ai-dynamo/dynamo/releases/latest)
22
[![Discord](https://dcbadge.limes.pink/api/server/D92uqZRjCZ?style=flat)](https://discord.gg/D92uqZRjCZ)
23
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ai-dynamo/dynamo)
Meenakshi Sharma's avatar
Meenakshi Sharma committed
24

25
| **[Roadmap](https://github.com/ai-dynamo/dynamo/issues/2486)** | **[Support matrix](https://github.com/ai-dynamo/dynamo/blob/main/docs/reference/support-matrix.md)** | **[Documentation](https://docs.nvidia.com/dynamo/latest/index.html)** | **[Examples](https://github.com/ai-dynamo/dynamo/tree/main/examples)** | **[Prebuilt containers](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo)** | **[Design Proposals](https://github.com/ai-dynamo/enhancements)** | **[Blogs](https://developer.nvidia.com/blog/tag/nvidia-dynamo)**
26

Anish's avatar
Anish committed
27
# NVIDIA Dynamo
28

Anish's avatar
Anish committed
29
High-throughput, low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.
30

31
32
## Latest News

33
- [08/05] Deploy `openai/gpt-oss-120b` with disaggregated serving on NVIDIA Blackwell GPUs using Dynamo [➡️ link](./docs/backends/trtllm/gpt-oss.md)
34

Anish's avatar
Anish committed
35
## The Era of Multi-GPU, Multi-Node
36

Anish's avatar
Anish committed
37
38
39
<p align="center">
  <img src="./docs/images/frontpage-gpu-vertical.png" alt="Multi Node Multi-GPU topology" width="600" />
</p>
40

Anish's avatar
Anish committed
41
Large language models are quickly outgrowing the memory and compute budget of any single GPU. Tensor-parallelism solves the capacity problem by spreading each layer across many GPUs—and sometimes many servers—but it creates a new one: how do you coordinate those shards, route requests, and share KV cache fast enough to feel like one accelerator? This orchestration gap is exactly what NVIDIA Dynamo is built to close.
42

Anish's avatar
Anish committed
43
Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLang or others) and captures LLM-specific capabilities such as:
44

Neelay Shah's avatar
Neelay Shah committed
45
46
47
48
49
- **Disaggregated prefill & decode inference** – Maximizes GPU throughput and facilitates trade off between throughput and latency.
- **Dynamic GPU scheduling** – Optimizes performance based on fluctuating demand
- **LLM-aware request routing** – Eliminates unnecessary KV cache re-computation
- **Accelerated data transfer** – Reduces inference response time using NIXL.
- **KV cache offloading** – Leverages multiple memory hierarchies for higher system throughput
50

Anish's avatar
Anish committed
51
52
53
54
55
<p align="center">
  <img src="./docs/images/frontpage-architecture.png" alt="Dynamo architecture" width="600" />
</p>

## Framework Support Matrix
Neelay Shah's avatar
Neelay Shah committed
56

57
58
| Feature                                                                                           | vLLM | SGLang | TensorRT-LLM |
| ------------------------------------------------------------------------------------------------- | ---- | ------ | ------------ |
59
60
61
| [**Disaggregated Serving**](/docs/design_docs/disagg_serving.md)                                 | ✅   | ✅     | ✅           |
| [**Conditional Disaggregation**](/docs/design_docs/disagg_serving.md#conditional-disaggregation) | 🚧   | 🚧     | 🚧           |
| [**KV-Aware Routing**](/docs/router/kv_cache_routing.md)                                    | ✅   | ✅     | ✅           |
62
63
64
| [**Load Based Planner**](docs/planner/load_planner.md)                                      | 🚧   | 🚧     | 🚧           |
| [**SLA-Based Planner**](docs/planner/sla_planner.md)                                        | ✅   | ✅     | ✅           |
| [**KVBM**](docs/kvbm/kvbm_architecture.md)                                               | ✅   | 🚧     | ✅           |
65

Anish's avatar
Anish committed
66
To learn more about each framework and their capabilities, check out each framework's README!
67

68
69
70
- **[vLLM](docs/backends/vllm/README.md)**
- **[SGLang](docs/backends/sglang/README.md)**
- **[TensorRT-LLM](docs/backends/trtllm/README.md)**
71

Anish's avatar
Anish committed
72
73
74
Built in Rust for performance and in Python for extensibility, Dynamo is fully open-source and driven by a transparent, OSS (Open Source Software) first development approach.

# Installation
75

Neelay Shah's avatar
Neelay Shah committed
76
The following examples require a few system level packages.
77
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [docs/reference/support-matrix.md](docs/reference/support-matrix.md)
78

Anish's avatar
Anish committed
79
80
81
## 1. Initial setup

The Dynamo team recommends the `uv` Python package manager, although any way works. Install uv:
82

Anish's avatar
Anish committed
83
84
85
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```
86

87
88
89
90
91
92
93
94
### Install Python development headers

Backend engines require Python development headers for JIT compilation. Install them with:

```bash
sudo apt install python3-dev
```

95
### Install etcd (optional) and NATS (required)
Anish's avatar
Anish committed
96

97
To coordinate across a data center, Dynamo relies on etcd and NATS. These will be used in production. To run Dynamo locally etcd is optional.
98

99
100
- [etcd](https://etcd.io/) can be run directly as `./etcd`.
- [nats](https://nats.io/) needs jetstream enabled: `nats-server -js`.
101

Anish's avatar
Anish committed
102
To quickly setup etcd & NATS, you can also run:
103

104
```bash
Anish's avatar
Anish committed
105
106
# At the root of the repository:
docker compose -f deploy/docker-compose.yml up -d
107
108
```

109
110
111
To run locally without etcd, pass `--store-kv file` to both the frontend and workers. The directory used for key-value data can be configured via the `DYN_FILE_KV` environment variable (example: `export DYN_FILE_KV=/data/kv/dynamo`). Defaults to `$TMPDIR/dynamo_store_kv`.


Anish's avatar
Anish committed
112
## 2. Select an engine
113

114
We publish Python wheels specialized for each of our supported engines: vllm, sglang, and trtllm. The examples that follow use SGLang; continue reading for other engines.
115

116
```
117
118
119
uv venv venv
source venv/bin/activate
uv pip install pip
120

121
# Choose one
Anish's avatar
Anish committed
122
uv pip install "ai-dynamo[sglang]"  #replace with [vllm], [trtllm], etc.
123
```
124

Anish's avatar
Anish committed
125
## 3. Run Dynamo
126

127
128
129
130
131
132
133
134
135
136
### Sanity check (optional)

Before trying out Dynamo, you can verify your system configuration and dependencies:

```bash
./deploy/sanity_check.py
```

This is a quick check for system resources, development tools, LLM frameworks, and Dynamo components.

137
138
139
### Running an LLM API server

Dynamo provides a simple way to spin up a local set of inference components including:
140

Neelay Shah's avatar
Neelay Shah committed
141
142
143
- **OpenAI Compatible Frontend** – High performance OpenAI compatible http api server written in Rust.
- **Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
- **Workers** – Set of pre-configured LLM serving engines.
144
145

```
Graham King's avatar
Graham King committed
146
147
# Start an OpenAI compatible HTTP server, a pre-processor (prompt templating and tokenization) and a router.
# Pass the TLS certificate and key paths to use HTTPS instead of HTTP.
148
149
# Pass --store-kv to use the filesystem instead of etcd. The workers and frontend must share a disk.
python -m dynamo.frontend --http-port 8000 [--tls-cert-path cert.pem] [--tls-key-path key.pem] [--store-kv file]
150

Anish's avatar
Anish committed
151
# Start the SGLang engine, connecting to NATS and etcd to receive requests. You can run several of these,
152
# both for the same model and for multiple models. The frontend node will discover them.
153
154
# Pass --store-kv to use the filesystem instead of etcd. The workers and frontend must share a disk.
python -m dynamo.sglang --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B [--store-kv file]
155
156
```

Neelay Shah's avatar
Neelay Shah committed
157
#### Send a Request
158

159
```bash
160
curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
Neelay Shah's avatar
Neelay Shah committed
161
162
163
164
165
166
167
168
169
170
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "messages": [
    {
        "role": "user",
        "content": "Hello, how are you?"
    }
    ],
    "stream":false,
    "max_tokens": 300
  }' | jq
171
```
172

173
174
Rerun with `curl -N` and change `stream` in the request to `true` to get the responses as soon as the engine issues them.

Anish's avatar
Anish committed
175
176
### Deploying Dynamo

177
- Follow the [Quickstart Guide](docs/kubernetes/README.md) to deploy on Kubernetes.
178
- Check out [Backends](examples/backends) to deploy various workflow configurations (e.g. SGLang with router, vLLM with disaggregated serving, etc.)
Anish's avatar
Anish committed
179
- Run some [Examples](examples) to learn about building components in Dynamo and exploring various integrations.
180

181
182
183
184
### Benchmarking Dynamo

Dynamo provides comprehensive benchmarking tools to evaluate and optimize your deployments:

185
- **[Benchmarking Guide](docs/benchmarks/benchmarking.md)** – Compare deployment topologies (aggregated vs. disaggregated vs. vanilla vLLM) using AIPerf
186
- **[SLA-Driven Dynamo Deployments](docs/planner/sla_planner_quickstart.md)** – Optimize your deployment to meet SLA requirements
187

Anish's avatar
Anish committed
188
# Engines
189

Anish's avatar
Anish committed
190
Dynamo is designed to be inference engine agnostic. To use any engine with Dynamo, NATS and etcd need to be installed, along with a Dynamo frontend (`python -m dynamo.frontend [--interactive]`).
191

Anish's avatar
Anish committed
192
## vLLM
193
194
195
196
197
198

```
uv pip install ai-dynamo[vllm]
```

Run the backend/worker like this:
199

200
201
202
203
```
python -m dynamo.vllm --help
```

Anish's avatar
Anish committed
204
vLLM attempts to allocate enough KV cache for the full context length at startup. If that does not fit in your available memory pass `--context-length <value>`.
205
206
207

To specify which GPUs to use set environment variable `CUDA_VISIBLE_DEVICES`.

Anish's avatar
Anish committed
208
## SGLang
209
210

```
211
212
213
# Install libnuma
apt install -y libnuma-dev

214
215
216
217
uv pip install ai-dynamo[sglang]
```

Run the backend/worker like this:
218

219
```
220
python -m dynamo.sglang --help
221
222
```

223
You can pass any sglang flags directly to this worker, see https://docs.sglang.ai/advanced_features/server_arguments.html . See there to use multiple GPUs.
224

Anish's avatar
Anish committed
225
## TensorRT-LLM
226

Anish's avatar
Anish committed
227
It is recommended to use [NGC PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for running the TensorRT-LLM engine.
228
229
230

> [!Note]
> Ensure that you select a PyTorch container image version that matches the version of TensorRT-LLM you are using.
231
> For example, if you are using `tensorrt-llm==1.1.0rc5`, use the PyTorch container image version `25.06`.
232
233
234
235
236
> To find the correct PyTorch container version for your desired `tensorrt-llm` release, visit the [TensorRT-LLM Dockerfile.multi](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docker/Dockerfile.multi) on GitHub. Switch to the branch that matches your `tensorrt-llm` version, and look for the `BASE_TAG` line to identify the recommended PyTorch container tag.

> [!Important]
> Launch container with the following additional settings `--shm-size=1g --ulimit memlock=-1`

Anish's avatar
Anish committed
237
### Install prerequisites
238

239
240
```
# Optional step: Only required for Blackwell and Grace Hopper
241
242
243
244
uv pip install torch==2.7.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

# Required until the trtllm version is bumped to include this pinned dependency itself
uv pip install "cuda-python>=12,<13"
245
246
247
248
249
250
251

sudo apt-get -y install libopenmpi-dev
```

> [!Tip]
> You can learn more about these prequisites and known issues with TensorRT-LLM pip based installation [here](https://nvidia.github.io/TensorRT-LLM/installation/linux.html).

Anish's avatar
Anish committed
252
### After installing the pre-requisites above, install Dynamo
253

254
```
255
uv pip install ai-dynamo[trtllm]
256
257
258
```

Run the backend/worker like this:
259

260
261
262
263
264
```
python -m dynamo.trtllm --help
```

To specify which GPUs to use set environment variable `CUDA_VISIBLE_DEVICES`.
265

Anish's avatar
Anish committed
266
# Developing Locally
267

Anish's avatar
Anish committed
268
## 1. Install libraries
269
270

**Ubuntu:**
271

272
273
274
```
sudo apt install -y build-essential libhwloc-dev libudev-dev pkg-config libclang-dev protobuf-compiler python3-dev cmake
```
275

276
**macOS:**
277

278
- [Homebrew](https://brew.sh/)
279

280
281
282
283
```
# if brew is not installed on your system, install it
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
284

285
- [Xcode](https://developer.apple.com/xcode/)
286

287
288
```
brew install cmake protobuf
289

290
291
292
293
## Check that Metal is accessible
xcrun -sdk macosx metal
```

294
If Metal is accessible, you should see an error like `metal: error: no input files`, which confirms it is installed correctly.
295

Anish's avatar
Anish committed
296
## 2. Install Rust
297

298
299
300
```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
301
```
302

Anish's avatar
Anish committed
303
## 3. Create a Python virtual env:
304

305
306
307
Follow the instructions in [uv installation](https://docs.astral.sh/uv/#installation) guide to install uv if you don't have `uv` installed. Once uv is installed, create a virtual environment and activate it.

- Install uv
308

309
310
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
311
```
312
313

- Create a virtual environment
314

315
```bash
316
317
318
uv venv dynamo
source dynamo/bin/activate
```
319

Anish's avatar
Anish committed
320
## 4. Install build tools
321

322
323
324
```
uv pip install pip maturin
```
325

326
[Maturin](https://github.com/PyO3/maturin) is the Rust<->Python bindings build tool.
327

Anish's avatar
Anish committed
328
## 5. Build the Rust bindings
329

330
```
331
cd lib/bindings/python
332
333
334
maturin develop --uv
```

Anish's avatar
Anish committed
335
## 6. Install the wheel
336
337
338

```
cd $PROJECT_ROOT
339
uv pip install -e .
340
341
342
343
```

You should now be able to run `python -m dynamo.frontend`.

344
Remember that nats and etcd must typically be running (see earlier).
345
346
347
348

Set the environment variable `DYN_LOG` to adjust the logging level; for example, `export DYN_LOG=debug`. It has the same syntax as `RUST_LOG`.

If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.