README.md 7.95 KB
Newer Older
Neelay Shah's avatar
Neelay Shah committed
1
<!--
Neelay Shah's avatar
Neelay Shah committed
2
SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Neelay Shah's avatar
Neelay Shah committed
3
SPDX-License-Identifier: Apache-2.0
4
5
6
7
8
9
10
11
12
13
14
15

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Neelay Shah's avatar
Neelay Shah committed
16
-->
17
![Dynamo banner](./docs/images/frontpage-banner.png)
Neelay Shah's avatar
Neelay Shah committed
18

19
20
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![GitHub Release](https://img.shields.io/github/v/release/ai-dynamo/dynamo)](https://github.com/ai-dynamo/dynamo/releases/latest)
21
[![Discord](https://dcbadge.limes.pink/api/server/D92uqZRjCZ?style=flat)](https://discord.gg/nvidia-dynamo)
Meenakshi Sharma's avatar
Meenakshi Sharma committed
22

23
| **[Roadmap](https://github.com/ai-dynamo/dynamo/issues/762)** | **[User Guides](https://docs.nvidia.com/dynamo/latest/index.html)** | **[Support Matrix](docs/support_matrix.md)** | **[Architecture and Features](docs/architecture/architecture.md)** | **[APIs](lib/bindings/python/README.md)** | **[SDK](deploy/dynamo/sdk/README.md)** |
Neelay Shah's avatar
Neelay Shah committed
24

25
26
### 📢 **Please join us for our** [ **first Dynamo in-person meetup with vLLM and SGLang leads**](https://events.nvidia.com/nvidiadynamousermeetups) **on 6/5 (Thu) in SF!** ###

27
28
29
30
31
32
33
34
35
36
37
38
39
40

### The Era of Multi-Node, Multi-GPU

![GPU Evolution](./docs/images/frontpage-gpu-evolution.png)


Large language models are quickly outgrowing the memory and compute budget of any single GPU. Tensor-parallelism solves the capacity problem by spreading each layer across many GPUs—and sometimes many servers—but it creates a new one: how do you coordinate those shards, route requests, and share KV cache fast enough to feel like one accelerator? This orchestration gap is exactly what NVIDIA Dynamo is built to close.

![Multi Node Multi-GPU topology](./docs/images/frontpage-gpu-vertical.png)



### Introducing NVIDIA Dynamo

Neelay Shah's avatar
Neelay Shah committed
41
NVIDIA Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments. Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLang or others) and captures LLM-specific capabilities such as:
42

43
44
![Dynamo architecture](./docs/images/frontpage-architecture.png)

Neelay Shah's avatar
Neelay Shah committed
45
46
47
48
49
- **Disaggregated prefill & decode inference** – Maximizes GPU throughput and facilitates trade off between throughput and latency.
- **Dynamic GPU scheduling** – Optimizes performance based on fluctuating demand
- **LLM-aware request routing** – Eliminates unnecessary KV cache re-computation
- **Accelerated data transfer** – Reduces inference response time using NIXL.
- **KV cache offloading** – Leverages multiple memory hierarchies for higher system throughput
50

Neelay Shah's avatar
Neelay Shah committed
51
Built in Rust for performance and in Python for extensibility, Dynamo is fully open-source and driven by a transparent, OSS (Open Source Software) first development approach.
Neelay Shah's avatar
Neelay Shah committed
52

53
54


Neelay Shah's avatar
Neelay Shah committed
55
### Installation
56

Neelay Shah's avatar
Neelay Shah committed
57
The following examples require a few system level packages.
58
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [docs/support_matrix.md](docs/support_matrix.md)
59

Neelay Shah's avatar
Neelay Shah committed
60
61
```
apt-get update
62
63
64
DEBIAN_FRONTEND=noninteractive apt-get install -yq python3-dev python3-pip python3-venv libucx0
python3 -m venv venv
source venv/bin/activate
65

66
pip install "ai-dynamo[all]"
Neelay Shah's avatar
Neelay Shah committed
67
```
68
69
> [!NOTE]
> To ensure compatibility, please refer to the examples in the release branch or tag that matches the version you installed.
70

71
### Building the Dynamo Base Image
72

73
74
75
76
Although not needed for local development, deploying your Dynamo pipelines to Kubernetes will require you to build and push a Dynamo base image to your container registry. You can use any container registry of your choice, such as:
- Docker Hub (docker.io)
- NVIDIA NGC Container Registry (nvcr.io)
- Any private registry
77

78
Here's how to build it:
79

80
```bash
81
82
83
84
./container/build.sh
docker tag dynamo:latest-vllm <your-registry>/dynamo-base:latest-vllm
docker login <your-registry>
docker push <your-registry>/dynamo-base:latest-vllm
85
86
```

87
88
89
90
Notes about builds for specific frameworks:
- For specific details on the `--framework vllm` build, see [here](examples/llm/README.md).
- For specific details on the `--framework tensorrtllm` build, see [here](examples/tensorrt_llm/README.md).

91
92
93
Note about AWS environments:
- If deploying Dynamo in AWS, make sure to build the container with EFA support using the `--make-efa` flag.

94
95
After building, you can use this image by setting the `DYNAMO_IMAGE` environment variable to point to your built image:
```bash
96
export DYNAMO_IMAGE=<your-registry>/dynamo-base:latest-vllm
97
```
98

99
100
101
> [!NOTE]
> We are working on leaner base images that can be built using the targets in the top-level Earthfile.

Neelay Shah's avatar
Neelay Shah committed
102
### Running and Interacting with an LLM Locally
103

Neelay Shah's avatar
Neelay Shah committed
104
105
106
To run a model and interact with it locally you can call `dynamo
run` with a hugging face model. `dynamo run` supports several backends
including: `mistralrs`, `sglang`, `vllm`, and `tensorrtllm`.
107

Neelay Shah's avatar
Neelay Shah committed
108
#### Example Command
109

Neelay Shah's avatar
Neelay Shah committed
110
111
```
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B
112
```
113

Neelay Shah's avatar
Neelay Shah committed
114
115
116
117
118
```
? User › Hello, how are you?
✔ User · Hello, how are you?
Okay, so I'm trying to figure out how to respond to the user's greeting. They said, "Hello, how are you?" and then followed it with "Hello! I'm just a program, but thanks for asking." Hmm, I need to come up with a suitable reply. ...
```
119

Neelay Shah's avatar
Neelay Shah committed
120
### LLM Serving
121

Neelay Shah's avatar
Neelay Shah committed
122
123
Dynamo provides a simple way to spin up a local set of inference
components including:
124

Neelay Shah's avatar
Neelay Shah committed
125
126
127
- **OpenAI Compatible Frontend** – High performance OpenAI compatible http api server written in Rust.
- **Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
- **Workers** – Set of pre-configured LLM serving engines.
128

Neelay Shah's avatar
Neelay Shah committed
129
130
To run a minimal configuration you can use a pre-configured
example.
131

Neelay Shah's avatar
Neelay Shah committed
132
#### Start Dynamo Distributed Runtime Services
133

Neelay Shah's avatar
Neelay Shah committed
134
First start the Dynamo Distributed Runtime services:
135
136

```bash
137
docker compose -f deploy/metrics/docker-compose.yml up -d
138
```
Neelay Shah's avatar
Neelay Shah committed
139
140
141
142
#### Start Dynamo LLM Serving Components

Next serve a minimal configuration with an http server, basic
round-robin router, and a single worker.
143
144

```bash
Neelay Shah's avatar
Neelay Shah committed
145
146
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
147
148
```

Neelay Shah's avatar
Neelay Shah committed
149
#### Send a Request
150

151
```bash
Neelay Shah's avatar
Neelay Shah committed
152
153
154
155
156
157
158
159
160
161
162
curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "messages": [
    {
        "role": "user",
        "content": "Hello, how are you?"
    }
    ],
    "stream":false,
    "max_tokens": 300
  }' | jq
163
```
164
165
166

### Local Development

167
If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.
168

169
Otherwise, to develop locally, we recommend working inside of the container
170
171
172
173
174
175

```bash
./container/build.sh
./container/run.sh -it --mount-workspace

cargo build --release
176
177
178
179
mkdir -p /workspace/deploy/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/http /workspace/deploy/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/llmctl /workspace/deploy/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/dynamo-run /workspace/deploy/sdk/src/dynamo/sdk/cli/bin
180
181

uv pip install -e .
182
export PYTHONPATH=$PYTHONPATH:/workspace/deploy/sdk/src:/workspace/components/planner/src
183
```
184

185

186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
#### Conda Environment

Alternately, you can use a conda environment

```bash
conda activate <ENV_NAME>

pip install nixl # Or install https://github.com/ai-dynamo/nixl from source

cargo build --release

# To install ai-dynamo-runtime from source
cd lib/bindings/python
pip install .

cd ../../../
202
pip install ".[all]"
203
204

# To test
205
docker compose -f deploy/metrics/docker-compose.yml up -d
206
207
208
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
```