Large language models exceed single-GPU capacity. Tensor parallelism spreads layers across GPUs but creates coordination challenges. Dynamo closes this orchestration gap.
Dynamo is inference engine agnostic (supports TRT-LLM, vLLM, SGLang) and provides:
Large language models are quickly outgrowing the memory and compute budget of any single GPU. Tensor-parallelism solves the capacity problem by spreading each layer across many GPUs—and sometimes many servers—but it creates a new one: how do you coordinate those shards, route requests, and share KV cache fast enough to feel like one accelerator? This orchestration gap is exactly what NVIDIA Dynamo is built to close.
Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLang or others) and captures LLM-specific capabilities such as:
| Path | Use Case | Time | Requirements |
|------|----------|------|--------------|
| [**Local Quick Start**](#local-quick-start) | Test on a single machine | ~5 min | 1 GPU, Ubuntu 24.04 |
| [**Kubernetes Deployment**](#kubernetes-deployment) | Production multi-node clusters | ~30 min | K8s cluster with GPUs |
-**Disaggregated prefill & decode inference** – Maximizes GPU throughput and facilitates trade off between throughput and latency.
## Contributing
-**Dynamic GPU scheduling** – Optimizes performance based on fluctuating demand
Built in Rust for performance and in Python for extensibility, Dynamo is fully open-source and driven by a transparent, OSS (Open Source Software) first development approach.
-**[Contributing Guide](CONTRIBUTING.md)** – How to get started
-**[Report a Bug](https://github.com/ai-dynamo/dynamo/issues/new?template=bug_report.yml)** – Found an issue?
-**[Feature Request](https://github.com/ai-dynamo/dynamo/issues/new?template=feature_request.yml)** – Have an idea?
# Installation
# Local Quick Start
The following examples require a few system level packages.
The following examples require a few system level packages.
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [docs/reference/support-matrix.md](docs/reference/support-matrix.md)
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [docs/reference/support-matrix.md](docs/reference/support-matrix.md)
## 1. Initial setup
## 1. Initial Setup
The Dynamo team recommends the `uv` Python package manager, although any way works. Install uv:
The Dynamo team recommends the `uv` Python package manager, although any way works. Install uv:
...
@@ -86,7 +101,7 @@ The Dynamo team recommends the `uv` Python package manager, although any way wor
...
@@ -86,7 +101,7 @@ The Dynamo team recommends the `uv` Python package manager, although any way wor
curl -LsSf https://astral.sh/uv/install.sh | sh
curl -LsSf https://astral.sh/uv/install.sh | sh
```
```
### Install Python development headers
### Install Python Development Headers
Backend engines require Python development headers for JIT compilation. Install them with:
Backend engines require Python development headers for JIT compilation. Install them with:
...
@@ -94,7 +109,7 @@ Backend engines require Python development headers for JIT compilation. Install
...
@@ -94,7 +109,7 @@ Backend engines require Python development headers for JIT compilation. Install
sudo apt install python3-dev
sudo apt install python3-dev
```
```
## 2. Select an engine
## 2. Select an Engine
We publish Python wheels specialized for each of our supported engines: vllm, sglang, and trtllm. The examples that follow use SGLang; continue reading for other engines.
We publish Python wheels specialized for each of our supported engines: vllm, sglang, and trtllm. The examples that follow use SGLang; continue reading for other engines.
...
@@ -109,17 +124,17 @@ uv pip install "ai-dynamo[sglang]" #replace with [vllm], [trtllm], etc.
...
@@ -109,17 +124,17 @@ uv pip install "ai-dynamo[sglang]" #replace with [vllm], [trtllm], etc.
## 3. Run Dynamo
## 3. Run Dynamo
### Sanity check (optional)
### Sanity Check (Optional)
Before trying out Dynamo, you can verify your system configuration and dependencies:
Before trying out Dynamo, you can verify your system configuration and dependencies:
```bash
```bash
./deploy/sanity_check.py
python3 deploy/sanity_check.py
```
```
This is a quick check for system resources, development tools, LLM frameworks, and Dynamo components.
This is a quick check for system resources, development tools, LLM frameworks, and Dynamo components.
### Running an LLM API server
### Running an LLM API Server
Dynamo provides a simple way to spin up a local set of inference components including:
Dynamo provides a simple way to spin up a local set of inference components including:
...
@@ -127,18 +142,18 @@ Dynamo provides a simple way to spin up a local set of inference components incl
...
@@ -127,18 +142,18 @@ Dynamo provides a simple way to spin up a local set of inference components incl
-**Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
-**Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
-**Workers** – Set of pre-configured LLM serving engines.
-**Workers** – Set of pre-configured LLM serving engines.
```
```bash
# Start an OpenAI compatible HTTP server with prompt templating, tokenization, and routing.
# Start an OpenAI compatible HTTP server with prompt templating, tokenization, and routing.
# Pass the TLS certificate and key paths to use HTTPS instead of HTTP.
# For local dev: --store-kv file avoids etcd (workers and frontend must share a disk)
# Pass --store-kv to use the filesystem instead of etcd. The workers and frontend must share a disk.
> **Note:** vLLM workers enable prefix caching by default, which requires NATS. For dependency-free local development with vLLM, add `--no-enable-prefix-caching`. See [Service Discovery and Messaging](#service-discovery-and-messaging) for details.
-**Benchmark**: Use [AIPerf](docs/benchmarks/benchmarking.md) to measure performance
-**Try other engines**: [vLLM](docs/backends/vllm/), [SGLang](docs/backends/sglang/), [TensorRT-LLM](docs/backends/trtllm/)
-**[Benchmarking Guide](docs/benchmarks/benchmarking.md)** – Compare deployment topologies (aggregated vs. disaggregated vs. vanilla vLLM) using AIPerf
# Kubernetes Deployment
-**[SLA-Driven Dynamo Deployments](docs/planner/sla_planner_quickstart.md)** – Optimize your deployment to meet SLA requirements
## Frontend OpenAPI specification
For production deployments on Kubernetes clusters with multiple GPUs.
The OpenAI-compatible HTTP frontend exposes an OpenAPI 3 specification at `/openapi.json`.
## Prerequisites
To generate and persist the same specification without running the server (for example for CI, documentation, or NIM integration), run:
```bash
- Kubernetes cluster with GPU nodes
cargo run -p dynamo-llm --bin generate-frontend-openapi
This writes the current frontend spec to `docs/frontends/openapi.json` at the repository root.
## Production Recipes
# Engines
Pre-built deployment configurations for common models and topologies:
Dynamo is designed to be inference engine agnostic. To use any engine with Dynamo, start a Dynamo frontend (`python -m dynamo.frontend`). For local development, pass `--store-kv file` to avoid etcd dependency. NATS is optional and only required for KV-aware routing.
See [recipes/README.md](recipes/README.md) for the full list and deployment instructions.
```
## Cloud Deployment Guides
uv pip install ai-dynamo[vllm]
```
Run the backend/worker like this:
-[Amazon EKS](examples/deployments/EKS/)
-[Google GKE](examples/deployments/GKE/)
```
# Concepts
python -m dynamo.vllm --help
```
vLLM attempts to allocate enough KV cache for the full context length at startup. If that does not fit in your available memory pass `--context-length <value>`.
## Engines
To specify which GPUs to use set environment variable `CUDA_VISIBLE_DEVICES`.
Dynamo is inference engine agnostic. Install the wheel for your chosen engine and run with `python3 -m dynamo.<engine> --help`.
> **Note:** TensorRT-LLM requires `pip` (not `uv`) due to URL-based dependencies. See the [TRT-LLM guide](docs/backends/trtllm/) for container setup and prerequisites.
# Install libnuma
apt install -y libnuma-dev
uv pip install ai-dynamo[sglang]
Use `CUDA_VISIBLE_DEVICES` to specify which GPUs to use. Engine-specific options (context length, multi-GPU, etc.) are documented in each backend guide.
```
Run the backend/worker like this:
## Service Discovery and Messaging
```
Dynamo uses TCP for inter-component communication. External services are optional for most deployments:
python -m dynamo.sglang --help
```
You can pass any sglang flags directly to this worker, see https://docs.sglang.ai/advanced_features/server_arguments.html . See there to use multiple GPUs.
| Deployment | etcd | NATS | Notes |
|------------|------|------|-------|
| **Kubernetes** | ❌ Not required | ❌ Not required | K8s-native discovery; TCP request plane |
| **Local development** | ❌ Not required | ❌ Not required | Pass `--store-kv file`; vLLM also needs `--no-enable-prefix-caching` |
For local development without external dependencies, pass `--store-kv file` (avoids etcd) to both the frontend and workers. vLLM users should also pass `--no-enable-prefix-caching` (avoids NATS); SGLang and TRT-LLM don't require this flag.
It is recommended to use [NGC PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for running the TensorRT-LLM engine.
For distributed non-Kubernetes deployments or KV-aware routing:
> [!Note]
-[etcd](https://etcd.io/) can be run directly as `./etcd`.
> Ensure that you select a PyTorch container image version that matches the version of TensorRT-LLM you are using.
> For example, if you are using `tensorrt-llm==1.2.0rc5`, use the PyTorch container image version `25.10`.
> To find the correct PyTorch container version for your desired `tensorrt-llm` release, visit the [TensorRT-LLM Dockerfile.multi](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docker/Dockerfile.multi) on GitHub. Switch to the branch that matches your `tensorrt-llm` version, and look for the `BASE_TAG` line to identify the recommended PyTorch container tag.
> [!Important]
To quickly setup both: `docker compose -f deploy/docker-compose.yml up -d`
> Launch container with the following additional settings `--shm-size=1g --ulimit memlock=-1`
### Install prerequisites
# Advanced Topics
```
## Benchmarking
# Optional step: Only required for non-container installations. The PyTorch 25.10 container already includes PyTorch 2.9.0 with CUDA 13.0.
# Optional step: Only required for disaggregated serving
-**[Benchmarking Guide](docs/benchmarks/benchmarking.md)** – Compare deployment topologies using AIPerf
sudo apt-get -y install libzmq3-dev
-**[SLA-Driven Deployments](docs/planner/sla_planner_quickstart.md)** – Optimize deployments to meet SLA requirements
```
> [!Tip]
## Frontend OpenAPI Specification
> You can learn more about these prequisites and known issues with TensorRT-LLM pip based installation [here](https://nvidia.github.io/TensorRT-LLM/installation/linux.html).
### After installing the pre-requisites above, install Dynamo
The OpenAI-compatible frontend exposes an OpenAPI 3 spec at `/openapi.json`. To generate without running the server:
```bash
cargo run -p dynamo-llm --bin generate-frontend-openapi
Follow the instructions in [uv installation](https://docs.astral.sh/uv/#installation) guide to install uv if you don't have `uv` installed. Once uv is installed, create a virtual environment and activate it.
Follow the instructions in [uv installation](https://docs.astral.sh/uv/#installation) guide to install uv if you don't have `uv` installed. Once uv is installed, create a virtual environment and activate it.
...
@@ -333,7 +316,7 @@ uv venv dynamo
...
@@ -333,7 +316,7 @@ uv venv dynamo
source dynamo/bin/activate
source dynamo/bin/activate
```
```
## 4. Install build tools
## 4. Install Build Tools
```
```
uv pip install pip maturin
uv pip install pip maturin
...
@@ -341,21 +324,21 @@ uv pip install pip maturin
...
@@ -341,21 +324,21 @@ uv pip install pip maturin
[Maturin](https://github.com/PyO3/maturin) is the Rust<->Python bindings build tool.
[Maturin](https://github.com/PyO3/maturin) is the Rust<->Python bindings build tool.
## 5. Build the Rust bindings
## 5. Build the Rust Bindings
```
```
cd lib/bindings/python
cd lib/bindings/python
maturin develop --uv
maturin develop --uv
```
```
## 6. Install the wheel
## 6. Install the Wheel
```
```
cd $PROJECT_ROOT
cd $PROJECT_ROOT
uv pip install -e .
uv pip install -e .
```
```
You should now be able to run `python -m dynamo.frontend`.
You should now be able to run `python3 -m dynamo.frontend`.
For local development, pass `--store-kv file` to avoid external dependencies (see Service Discovery and Messaging section).
For local development, pass `--store-kv file` to avoid external dependencies (see Service Discovery and Messaging section).
...
@@ -363,23 +346,6 @@ Set the environment variable `DYN_LOG` to adjust the logging level; for example,
...
@@ -363,23 +346,6 @@ Set the environment variable `DYN_LOG` to adjust the logging level; for example,
If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.
If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.
# Contributing
We welcome contributions! Whether it's bug reports, documentation improvements, or code contributions—every bit helps.
-**[Contributing Guide](CONTRIBUTING.md)** – How to get started
-**[Report a Bug](https://github.com/ai-dynamo/dynamo/issues/new?template=bug_report.yml)** – Found an issue?
-**[Feature Request](https://github.com/ai-dynamo/dynamo/issues/new?template=feature_request.yml)** – Have an idea?
Dan Gil <dagil@nvidia.com>