Large language models exceed single-GPU capacity. Tensor parallelism spreads layers across GPUs but creates coordination challenges. Dynamo closes this orchestration gap.
Dynamo is inference engine agnostic (supports TRT-LLM, vLLM, SGLang) and provides:
Large language models are quickly outgrowing the memory and compute budget of any single GPU. Tensor-parallelism solves the capacity problem by spreading each layer across many GPUs—and sometimes many servers—but it creates a new one: how do you coordinate those shards, route requests, and share KV cache fast enough to feel like one accelerator? This orchestration gap is exactly what NVIDIA Dynamo is built to close.
## Get Started
Dynamo is designed to be inference engine agnostic (supports TRT-LLM, vLLM, SGLang or others) and captures LLM-specific capabilities such as:
| Path | Use Case | Time | Requirements |
|------|----------|------|--------------|
| [**Local Quick Start**](#local-quick-start) | Test on a single machine | ~5 min | 1 GPU, Ubuntu 24.04 |
| [**Kubernetes Deployment**](#kubernetes-deployment) | Production multi-node clusters | ~30 min | K8s cluster with GPUs |
-**Disaggregated prefill & decode inference** – Maximizes GPU throughput and facilitates trade off between throughput and latency.
-**Dynamic GPU scheduling** – Optimizes performance based on fluctuating demand
Want to help shape the future of distributed LLM inference? We welcome contributors at all levels—from doc fixes to new features.
Built in Rust for performance and in Python for extensibility, Dynamo is fully open-source and driven by a transparent, OSS (Open Source Software) first development approach.
-**[Contributing Guide](CONTRIBUTING.md)** – How to get started
-**[Report a Bug](https://github.com/ai-dynamo/dynamo/issues/new?template=bug_report.yml)** – Found an issue?
-**[Feature Request](https://github.com/ai-dynamo/dynamo/issues/new?template=feature_request.yml)** – Have an idea?
# Installation
# Local Quick Start
The following examples require a few system level packages.
Recommended to use Ubuntu 24.04 with a x86_64 CPU. See [docs/reference/support-matrix.md](docs/reference/support-matrix.md)
## 1. Initial setup
## 1. Initial Setup
The Dynamo team recommends the `uv` Python package manager, although any way works. Install uv:
...
...
@@ -86,7 +101,7 @@ The Dynamo team recommends the `uv` Python package manager, although any way wor
curl -LsSf https://astral.sh/uv/install.sh | sh
```
### Install Python development headers
### Install Python Development Headers
Backend engines require Python development headers for JIT compilation. Install them with:
...
...
@@ -94,7 +109,7 @@ Backend engines require Python development headers for JIT compilation. Install
sudo apt install python3-dev
```
## 2. Select an engine
## 2. Select an Engine
We publish Python wheels specialized for each of our supported engines: vllm, sglang, and trtllm. The examples that follow use SGLang; continue reading for other engines.
...
...
@@ -109,17 +124,17 @@ uv pip install "ai-dynamo[sglang]" #replace with [vllm], [trtllm], etc.
## 3. Run Dynamo
### Sanity check (optional)
### Sanity Check (Optional)
Before trying out Dynamo, you can verify your system configuration and dependencies:
```bash
./deploy/sanity_check.py
python3 deploy/sanity_check.py
```
This is a quick check for system resources, development tools, LLM frameworks, and Dynamo components.
### Running an LLM API server
### Running an LLM API Server
Dynamo provides a simple way to spin up a local set of inference components including:
...
...
@@ -127,18 +142,18 @@ Dynamo provides a simple way to spin up a local set of inference components incl
-**Basic and Kv Aware Router** – Route and load balance traffic to a set of workers.
-**Workers** – Set of pre-configured LLM serving engines.
```
```bash
# Start an OpenAI compatible HTTP server with prompt templating, tokenization, and routing.
# Pass the TLS certificate and key paths to use HTTPS instead of HTTP.
# Pass --store-kv to use the filesystem instead of etcd. The workers and frontend must share a disk.
> **Note:** vLLM workers enable prefix caching by default, which requires NATS. For dependency-free local development with vLLM, add `--no-enable-prefix-caching`. See [Service Discovery and Messaging](#service-discovery-and-messaging) for details.
-**Benchmark**: Use [AIPerf](docs/benchmarks/benchmarking.md) to measure performance
-**Try other engines**: [vLLM](docs/backends/vllm/), [SGLang](docs/backends/sglang/), [TensorRT-LLM](docs/backends/trtllm/)
-**[Benchmarking Guide](docs/benchmarks/benchmarking.md)** – Compare deployment topologies (aggregated vs. disaggregated vs. vanilla vLLM) using AIPerf
-**[SLA-Driven Dynamo Deployments](docs/planner/sla_planner_quickstart.md)** – Optimize your deployment to meet SLA requirements
# Kubernetes Deployment
## Frontend OpenAPI specification
For production deployments on Kubernetes clusters with multiple GPUs.
The OpenAI-compatible HTTP frontend exposes an OpenAPI 3 specification at `/openapi.json`.
To generate and persist the same specification without running the server (for example for CI, documentation, or NIM integration), run:
## Prerequisites
```bash
cargo run -p dynamo-llm --bin generate-frontend-openapi
This writes the current frontend spec to `docs/frontends/openapi.json` at the repository root.
## Production Recipes
# Engines
Pre-built deployment configurations for common models and topologies:
Dynamo is designed to be inference engine agnostic. To use any engine with Dynamo, start a Dynamo frontend (`python -m dynamo.frontend`). For local development, pass `--store-kv file` to avoid etcd dependency. NATS is optional and only required for KV-aware routing.
See [recipes/README.md](recipes/README.md) for the full list and deployment instructions.
```
uv pip install ai-dynamo[vllm]
```
## Cloud Deployment Guides
Run the backend/worker like this:
-[Amazon EKS](examples/deployments/EKS/)
-[Google GKE](examples/deployments/GKE/)
```
python -m dynamo.vllm --help
```
# Concepts
vLLM attempts to allocate enough KV cache for the full context length at startup. If that does not fit in your available memory pass `--context-length <value>`.
## Engines
To specify which GPUs to use set environment variable `CUDA_VISIBLE_DEVICES`.
Dynamo is inference engine agnostic. Install the wheel for your chosen engine and run with `python3 -m dynamo.<engine> --help`.
> **Note:** TensorRT-LLM requires `pip` (not `uv`) due to URL-based dependencies. See the [TRT-LLM guide](docs/backends/trtllm/) for container setup and prerequisites.
uv pip install ai-dynamo[sglang]
```
Use `CUDA_VISIBLE_DEVICES` to specify which GPUs to use. Engine-specific options (context length, multi-GPU, etc.) are documented in each backend guide.
Run the backend/worker like this:
## Service Discovery and Messaging
```
python -m dynamo.sglang --help
```
Dynamo uses TCP for inter-component communication. External services are optional for most deployments:
You can pass any sglang flags directly to this worker, see https://docs.sglang.ai/advanced_features/server_arguments.html . See there to use multiple GPUs.
| Deployment | etcd | NATS | Notes |
|------------|------|------|-------|
| **Kubernetes** | ❌ Not required | ❌ Not required | K8s-native discovery; TCP request plane |
| **Local development** | ❌ Not required | ❌ Not required | Pass `--store-kv file`; vLLM also needs `--no-enable-prefix-caching` |
For local development without external dependencies, pass `--store-kv file` (avoids etcd) to both the frontend and workers. vLLM users should also pass `--no-enable-prefix-caching` (avoids NATS); SGLang and TRT-LLM don't require this flag.
It is recommended to use [NGC PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) for running the TensorRT-LLM engine.
For distributed non-Kubernetes deployments or KV-aware routing:
> [!Note]
> Ensure that you select a PyTorch container image version that matches the version of TensorRT-LLM you are using.
> For example, if you are using `tensorrt-llm==1.2.0rc5`, use the PyTorch container image version `25.10`.
> To find the correct PyTorch container version for your desired `tensorrt-llm` release, visit the [TensorRT-LLM Dockerfile.multi](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docker/Dockerfile.multi) on GitHub. Switch to the branch that matches your `tensorrt-llm` version, and look for the `BASE_TAG` line to identify the recommended PyTorch container tag.
-[etcd](https://etcd.io/) can be run directly as `./etcd`.
# Optional step: Only required for disaggregated serving
sudo apt-get -y install libzmq3-dev
```
-**[Benchmarking Guide](docs/benchmarks/benchmarking.md)** – Compare deployment topologies using AIPerf
-**[SLA-Driven Deployments](docs/planner/sla_planner_quickstart.md)** – Optimize deployments to meet SLA requirements
> [!Tip]
> You can learn more about these prequisites and known issues with TensorRT-LLM pip based installation [here](https://nvidia.github.io/TensorRT-LLM/installation/linux.html).
## Frontend OpenAPI Specification
### After installing the pre-requisites above, install Dynamo
The OpenAI-compatible frontend exposes an OpenAPI 3 spec at `/openapi.json`. To generate without running the server:
```bash
cargo run -p dynamo-llm --bin generate-frontend-openapi
Follow the instructions in [uv installation](https://docs.astral.sh/uv/#installation) guide to install uv if you don't have `uv` installed. Once uv is installed, create a virtual environment and activate it.
...
...
@@ -333,7 +316,7 @@ uv venv dynamo
source dynamo/bin/activate
```
## 4. Install build tools
## 4. Install Build Tools
```
uv pip install pip maturin
...
...
@@ -341,21 +324,21 @@ uv pip install pip maturin
[Maturin](https://github.com/PyO3/maturin) is the Rust<->Python bindings build tool.
## 5. Build the Rust bindings
## 5. Build the Rust Bindings
```
cd lib/bindings/python
maturin develop --uv
```
## 6. Install the wheel
## 6. Install the Wheel
```
cd $PROJECT_ROOT
uv pip install -e .
```
You should now be able to run `python -m dynamo.frontend`.
You should now be able to run `python3 -m dynamo.frontend`.
For local development, pass `--store-kv file` to avoid external dependencies (see Service Discovery and Messaging section).
...
...
@@ -363,23 +346,6 @@ Set the environment variable `DYN_LOG` to adjust the logging level; for example,
If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.
# Contributing
We welcome contributions! Whether it's bug reports, documentation improvements, or code contributions—every bit helps.
-**[Contributing Guide](CONTRIBUTING.md)** – How to get started
-**[Report a Bug](https://github.com/ai-dynamo/dynamo/issues/new?template=bug_report.yml)** – Found an issue?
-**[Feature Request](https://github.com/ai-dynamo/dynamo/issues/new?template=feature_request.yml)** – Have an idea?
Dan Gil <dagil@nvidia.com>