README.md 13.9 KB
Newer Older
1
2
3
4
5
<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

6
# Dynamo Production-Ready Recipes
7

8
Production-tested Kubernetes deployment recipes for LLM inference using NVIDIA Dynamo.
9

10
> **Prerequisites:** This guide assumes you have already installed the Dynamo Kubernetes Platform.
11
> If not, follow the **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** first.
12

13
## Available Recipes
14

15
### Feature Comparison Recipes
16

17
These recipes compare Dynamo performance features with benchmark results, each including both baseline and optimized deployment configurations:
18
19
20

| Model | Framework | Configuration | GPUs | Features |
|-------|-----------|---------------|------|----------|
21
22
23
| **[Qwen3-32B](qwen3-32b/)** | vLLM | Disagg + KV-Router | 16x H200 | **Disaggregated Serving + KV-Aware Routing** — benchmark comparison with real-world Mooncake traces |
| **[DeepSeek-V3.2-NVFP4](deepseek-v32-fp4/)** | TensorRT-LLM | Agg + Disagg WideEP | 32x GB200 | **Disaggregated Serving + KV-Aware Routing** — benchmark comparison with Mooncake-based synthetic coding trace |
| **[Qwen3-VL-30B-A3B-FP8](qwen3-vl-30b/)** | vLLM | Agg + Embedding Cache | 1x GB200 | **Multimodal Embedding Cache** — benchmark comparison showing +16% throughput, -28% TTFT |
24
25
26
27
28
29
30

### Aggregated & Disaggregated Recipes

These recipes demonstrate aggregated or disaggregated serving:

**GAIE Column**: Indicates whether the recipe includes integration with the [Gateway API Inference Extension (GAIE)](../deploy/inference-gateway/README.md) — a Kubernetes SIG project that extends the Gateway API for AI inference workloads, providing load balancing, model routing, and request management.

31
32
| Model | Framework | Mode | GPUs | Deployment | Benchmark | Notes | GAIE |
|-------|-----------|------|------|------------|-----------|-------|------|
33
| **[Llama-3-70B](llama-3-70b/vllm/agg/)** | vLLM | Aggregated | 4x H100/H200 | ✅ | ✅ | FP8 dynamic quantization | ✅ |
34
35
| **[Llama-3-70B](llama-3-70b/vllm/disagg-single-node/)** | vLLM | Disagg (Single-Node) | 8x H100/H200 | ✅ | ✅ | Prefill + Decode separation | ❌ |
| **[Llama-3-70B](llama-3-70b/vllm/disagg-multi-node/)** | vLLM | Disagg (Multi-Node) | 16x H100/H200 | ✅ | ✅ | 2 nodes, 8 GPUs each | ❌ |
36
37
38
39
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/agg/)** | TensorRT-LLM | Aggregated | 2x H100/H200/A100 | ✅ | ✅ | FP8 quantization | ❌ |
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 8x H100/H200/A100 | ✅ | ✅ | Prefill + Decode separation | ❌ |
| **[Qwen3-235B-A22B-FP8](qwen3-235b-a22b-fp8/trtllm/agg/)** | TensorRT-LLM | Aggregated | 16x H100/H200 | ✅ | ✅ | MoE model, TP4×EP4 | ❌ |
| **[Qwen3-235B-A22B-FP8](qwen3-235b-a22b-fp8/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 16x H100/H200 | ✅ | ✅ | MoE model, Prefill + Decode | ❌ |
40
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/agg/)** | TensorRT-LLM | Aggregated | 4x GB200 | ✅ | ✅ | Blackwell only, WideEP | ❌ |
41
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 5x Blackwell (GB200/B200) | ✅ | ✅ | Prefill/Decode split | ❌ |
42
43
44
45
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-8gpu/)** | SGLang | Disagg WideEP | 16x H200 | ✅ | ❌ | TP=8, single-node. Use `model-download-sglang.yaml` | ❌ |
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-16gpu/)** | SGLang | Disagg WideEP | 32x H200 | ✅ | ❌ | TP=16, multi-node. Use `model-download-sglang.yaml` | ❌ |
| **[DeepSeek-R1](deepseek-r1/trtllm/disagg/wide_ep/gb200/)** | TensorRT-LLM | Disagg WideEP (GB200) | 36x GB200 | ✅ | ✅ | Multi-node: 8 decode + 1 prefill nodes | ❌ |
| **[DeepSeek-R1](deepseek-r1/)** | vLLM | Disagg DEP16 | 32x H200 | ✅ | ❌ | Multi-node, data-expert parallel | ❌ |
46

47
48
49
**Legend:**
- **Deployment**: ✅ = Complete `deploy.yaml` manifest available
- **Benchmark**: ✅ = Includes `perf.yaml` for running AIPerf benchmarks
50

51
### Functional Recipes (Not Yet Benchmarked)
52

53
These recipes demonstrate functional deployments with Dynamo features, but have not yet been performance-tuned or paired with benchmark manifests.
54
55
56
57
58
59
60

| Model | Framework | Mode | GPUs | Deployment | Notes |
|-------|-----------|-------|------|------------|-------|
| **[Nemotron-3-Super-FP8](nemotron-3-super-fp8/vllm/agg/)** | vLLM | Aggregated | 4x H100/H200 | ✅ | TP=4, KV-aware routing |
| **[Nemotron-3-Super-FP8](nemotron-3-super-fp8/sglang/agg/)** | SGLang | Aggregated | 4x H100/H200 | ✅ | TP=4, KV-aware routing, 1.0+ |
| **[Nemotron-3-Super-FP8](nemotron-3-super-fp8/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 4x H100/H200 | ✅ | TP=2 prefill/decode split, UCX KV transfer |
| **[Nemotron-3-Super-FP8](nemotron-3-super-fp8/sglang/disagg/)** | SGLang | Disaggregated | 4x H100/H200 | ✅ | TP=2 prefill/decode split, nixl KV transfer, 1.0+ |
61
62
63
64
65
66
67
68
| **[Kimi-K2.5 (Baseten)](kimi-k2.5/trtllm/agg/baseten/)** | TensorRT-LLM | Aggregated | 8x B200 | ✅ | Text only — MoE model, TP8×EP8, reasoning + tool calling |

### Experimental Recipes

These recipes are under active development and may require additional setup steps (e.g., container patching). They are functional but not yet fully validated for production use.

| Model | Framework | Mode | GPUs | Deployment | Notes |
|-------|-----------|------|------|------------|-------|
69
| **[GLM-5-NVFP4](glm-5-nvfp4/sglang/disagg/)** | SGLang | Disagg Prefill/Decode | 20x GB200 | ✅ | NVFP4, EAGLE speculative decoding, TP16 decode + TP4 prefill. Requires [custom container build](glm-5-nvfp4/). |
70
| **[nvidia/Kimi-K2.5-NVFP4](kimi-k2.5/trtllm/agg/nvidia/)** | TensorRT-LLM | Aggregated | 8x B200 | ✅ | Text only — MoE model, TP8×EP8, reasoning + tool calling. Vision input not yet functional. |
71

72
## Recipe Structure
73

74
Each complete recipe follows this standard structure:
75

76
```
77
<model-name>/
78
├── README.md (optional)           # Model-specific deployment notes
79
├── model-cache/
80
81
82
83
84
85
│   ├── model-cache.yaml          # PersistentVolumeClaim for model storage
│   └── model-download.yaml       # Job to download model from HuggingFace
└── <framework>/                  # vllm, sglang, or trtllm
    └── <deployment-mode>/        # agg, disagg, disagg-single-node, etc.
        ├── deploy.yaml           # Complete DynamoGraphDeployment manifest
        └── perf.yaml (optional)  # AIPerf benchmark job
86
87
88
89
```

## Quick Start

90
### Prerequisites
91

92
**1. Dynamo Platform Installed**
93

94
The recipes require the Dynamo Kubernetes Platform to be installed. Follow the installation guide:
95

96
97
- **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** - Quickstart (~10 minutes)
- **[Detailed Installation Guide](../docs/kubernetes/installation-guide.md)** - Advanced options
98

99
**2. GPU Cluster Requirements**
100

101
102
Ensure your cluster has:
- GPU nodes matching recipe requirements (see table above)
103
- GPU operator installed
104
- Appropriate GPU drivers and container runtime
105

106
**3. HuggingFace Access**
107

108
Configure authentication to download models:
109
110

```bash
111
112
export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
113

114
115
116
117
# Create HuggingFace token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token-here" \
  -n ${NAMESPACE}
118
119
```

120
121
122
**4. Storage Configuration**

Update the `storageClassName` in `<model>/model-cache/model-cache.yaml` to match your cluster:
123
124

```bash
125
# Find your storage class name
126
kubectl get storageclass
127

128
129
130
# Edit the model-cache.yaml file and update:
# spec:
#   storageClassName: "your-actual-storage-class"
131
132
```

133
### Deploy a Recipe
134

135
**Step 1: Download Model**
136
137

```bash
138
cd recipes
139
140
# Update storageClassName in model-cache.yaml first!
kubectl apply -f <model>/model-cache/ -n ${NAMESPACE}
141

142
143
# Wait for download to complete (may take 10-60 minutes depending on model size)
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s
144

145
146
147
# Monitor progress
kubectl logs -f job/model-download -n ${NAMESPACE}
```
148

149
**Step 2: Deploy Service**
150

151
152
Update the image in `<model>/<framework>/<mode>/deploy.yaml`.

153
```bash
154
kubectl apply -f <model>/<framework>/<mode>/deploy.yaml -n ${NAMESPACE}
155

156
157
# Check deployment status
kubectl get dynamographdeployment -n ${NAMESPACE}
158

159
160
# Check pod status
kubectl get pods -n ${NAMESPACE}
161

162
163
# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l nvidia.com/dynamo-graph-deployment-name=<deployment-name> -n ${NAMESPACE} --timeout=600s
164
```
165

166
**Step 3: Test Deployment**
167

168
169
170
```bash
# Port forward to access the service locally
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
171

172
173
# In another terminal, test the endpoint
curl http://localhost:8000/v1/models
174

175
176
177
178
179
180
181
182
# Send a test request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'
183
```
184

185
**Step 4: Run Benchmark (Optional)**
186
187

```bash
188
189
# Only if perf.yaml exists in the recipe directory
kubectl apply -f <model>/<framework>/<mode>/perf.yaml -n ${NAMESPACE}
190

191
192
# Monitor benchmark progress
kubectl logs -f job/<benchmark-job-name> -n ${NAMESPACE}
193

194
195
# View results after completion
kubectl logs job/<benchmark-job-name> -n ${NAMESPACE} | tail -50
196
197
198
```


199
## Example Deployments
200

201
### Llama-3-70B with vLLM (Aggregated)
202
203

```bash
204
205
export NAMESPACE=dynamo-demo
kubectl create namespace ${NAMESPACE}
206

207
208
209
210
# Create HF token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}
211

212
# Deploy
213
cd recipes
214
215
216
217
218
219
kubectl apply -f llama-3-70b/model-cache/ -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s
kubectl apply -f llama-3-70b/vllm/agg/deploy.yaml -n ${NAMESPACE}

# Test
kubectl port-forward svc/llama3-70b-agg-frontend 8000:8000 -n ${NAMESPACE}
220
221
```

222
### Inference Gateway (GAIE) Integration (Optional)
atchernych's avatar
atchernych committed
223
224
225

For Llama-3-70B with vLLM (Aggregated), an example of integration with the Inference Gateway is provided.

226
227
228
229
First, deploy the Dynamo Graph per instructions above.

Then follow [Deploy Inference Gateway Section 2](../deploy/inference-gateway/README.md#2-deploy-inference-gateway) to install GAIE.

230
Update the containers.epp.image in the deployment file, i.e. llama-3-70b/vllm/agg/gaie/k8s-manifests/epp/deployment.yaml. It should match the release tag and be in the format `nvcr.io/nvidia/ai-dynamo/frontend:<version>` e.g. `nvcr.io/nvidia/ai-dynamo/frontend:0.9.0`
231
232
233
234
235
The recipe assumes you are using Kubernetes discovery backend and sets the `DYN_DISCOVERY_BACKEND` env variable in the epp deployment. If you want to use etcd enable the lines below and remove the DYN_DISCOVERY_BACKEND env var.
```bash
- name: ETCD_ENDPOINTS
  value: "dynamo-platform-etcd.$(PLATFORM_NAMESPACE):2379" #  update dynamo-platform to appropriate namespace
```
atchernych's avatar
atchernych committed
236
237
238

```bash
export DEPLOY_PATH=llama-3-70b/vllm/agg/
239
# DEPLOY_PATH=<model>/<framework>/<mode>/
atchernych's avatar
atchernych committed
240
kubectl apply -R -f "$DEPLOY_PATH/gaie/k8s-manifests" -n "$NAMESPACE"
atchernych's avatar
atchernych committed
241
```
atchernych's avatar
atchernych committed
242

243
### DeepSeek-R1 on GB200 (Multi-node)
244

245
See [deepseek-r1/trtllm/disagg/wide_ep/gb200/deploy.yaml](deepseek-r1/trtllm/disagg/wide_ep/gb200/deploy.yaml) for the complete multi-node WideEP configuration.
246

247
## Customization
248

249
250
251
252
253
Each `deploy.yaml` contains:
- **ConfigMap**: Engine-specific configuration (embedded in the manifest)
- **DynamoGraphDeployment**: Kubernetes resource definitions
- **Resource limits**: GPU count, memory, CPU requests/limits
- **Image references**: Container images with version tags
254

255
### Key Customization Points
256

257
258
259
260
261
262
**Model Configuration:**
```yaml
# In deploy.yaml under worker args:
args:
  - python3 -m dynamo.vllm --model <your-model-path> --served-model-name <name>
```
263

264
265
266
267
268
269
270
271
**GPU Resources:**
```yaml
resources:
  limits:
    gpu: "4"  # Adjust based on your requirements
  requests:
    gpu: "4"
```
272

273
274
275
276
277
278
**Scaling:**
```yaml
services:
  VllmDecodeWorker:
    replicas: 2  # Scale to multiple workers
```
279

280
281
282
283
284
285
286
**Router Mode:**
```yaml
# In Frontend args:
args:
  - python3 -m dynamo.frontend --router-mode kv --http-port 8000
# Options: round-robin, kv (KV-aware routing)
```
287

288
289
290
291
**Container Images:**
```yaml
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:x.y.z
# Update version tag as needed
292
293
```

294
## Troubleshooting
295

296
### Common Issues
297

298
299
300
301
**Pods stuck in Pending:**
- Check GPU availability: `kubectl describe node <node-name>`
- Verify storage class exists: `kubectl get storageclass`
- Check resource requests vs. available resources
302

303
304
305
306
**Model download fails:**
- Verify HuggingFace token is correct
- Check network connectivity from cluster
- Review job logs: `kubectl logs job/model-download -n ${NAMESPACE}`
307

308
309
310
311
**Workers fail to start:**
- Check GPU compatibility (driver version, CUDA version)
- Verify image pull secrets if using private registries
- Review pod logs: `kubectl logs <pod-name> -n ${NAMESPACE}`
312

313
**For more troubleshooting:**
314
315
- [Kubernetes Deployment Guide](../docs/kubernetes/README.md#troubleshooting)
- [Observability Documentation](../docs/kubernetes/observability/)
316

317
## Related Documentation
318

319
320
321
322
323
324
325
- **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** - Platform installation and concepts
- **[API Reference](../docs/kubernetes/api-reference.md)** - DynamoGraphDeployment CRD specification
- **[vLLM Backend Guide](../docs/backends/vllm/README.md)** - vLLM-specific features
- **[SGLang Backend Guide](../docs/backends/sglang/README.md)** - SGLang-specific features
- **[TensorRT-LLM Backend Guide](../docs/backends/trtllm/README.md)** - TensorRT-LLM features
- **[Observability](../docs/kubernetes/observability/)** - Monitoring and logging
- **[Benchmarking Guide](../docs/benchmarks/benchmarking.md)** - Performance testing
326

327
## Contributing
328

329
330
331
332
333
We welcome contributions of new recipes! See [CONTRIBUTING.md](CONTRIBUTING.md) for:
- Recipe submission guidelines
- Required components checklist
- Testing and validation requirements
- Documentation standards
334

335
336
337
338
339
340
341
342
### Recipe Quality Standards

A production-ready recipe must include:
- ✅ Complete `deploy.yaml` with DynamoGraphDeployment
- ✅ Model cache PVC and download job
- ✅ Benchmark recipe (`perf.yaml`) for performance testing
- ✅ Verification on target hardware
- ✅ Documentation of GPU requirements