README.md 10.7 KB
Newer Older
1
# Dynamo Production-Ready Recipes
2

3
Production-tested Kubernetes deployment recipes for LLM inference using NVIDIA Dynamo.
4

5
6
> **Prerequisites:** This guide assumes you have already installed the Dynamo Kubernetes Platform.
> If not, follow the **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** first.
7

8
## Available Recipes
9

10
11
12
13
14
15
16
17
18
| Model | Framework | Mode | GPUs | Deployment | Benchmark Recipe | Notes |GAIE integration |
|-------|-----------|------|------|------------|------------------|-------|------------------|
| **[Llama-3-70B](llama-3-70b/vllm/agg/)** | vLLM | Aggregated | 4x H100/H200 | ✅ | ✅ | FP8 dynamic quantization | ✅ | ❌ |
| **[Llama-3-70B](llama-3-70b/vllm/disagg-single-node/)** | vLLM | Disagg (Single-Node) | 8x H100/H200 | ✅ | ✅ | Prefill + Decode separation | ❌ |
| **[Llama-3-70B](llama-3-70b/vllm/disagg-multi-node/)** | vLLM | Disagg (Multi-Node) | 16x H100/H200 | ✅ | ✅ | 2 nodes, 8 GPUs each | ❌ |
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/agg/)** | TensorRT-LLM | Aggregated | 4x GPU | ✅ | ✅ | FP8 quantization | ❌ |
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 8x GPU | ✅ | ✅ | Prefill + Decode separation | ❌ |
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/agg/)** | TensorRT-LLM | Aggregated | 4x GB200 | ✅ | ✅ | Blackwell only, WideEP | ❌ |
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | TBD | ❌ | ❌ | Engine configs only, no K8s manifest | ❌ |
19
20
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-8gpu/)** | SGLang | Disagg WideEP | 8x H200 | ✅*1 | ❌ | Benchmark recipe pending | ❌ |
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-16gpu/)** | SGLang | Disagg WideEP | 16x H200 | ✅*1 | ❌ | Benchmark recipe pending | ❌ |
21
| **[DeepSeek-R1](deepseek-r1/trtllm/disagg/wide_ep/gb200/)** | TensorRT-LLM | Disagg WideEP (GB200) | 32+4 GB200 | ✅ | ✅ |Multi-node: 8 decode + 1 prefill nodes | ❌ |
22

23
24
*1: Please use `deepseek-r1/model-cache/model-download-sglang.yaml` to download the model into the PVC.

25
**Legend:**
26
27
28
29
- **Deployment**: ✅ = Complete `deploy.yaml` manifest available | ❌ = Missing or incomplete
- **Benchmark Recipe**: ✅ = Includes `perf.yaml` for running AIPerf benchmarks | ❌ = No benchmark recipe provided

## Recipe Structure
30

31
Each complete recipe follows this standard structure:
32

33
```
34
<model-name>/
35
├── README.md (optional)           # Model-specific deployment notes
36
├── model-cache/
37
38
39
40
41
42
│   ├── model-cache.yaml          # PersistentVolumeClaim for model storage
│   └── model-download.yaml       # Job to download model from HuggingFace
└── <framework>/                  # vllm, sglang, or trtllm
    └── <deployment-mode>/        # agg, disagg, disagg-single-node, etc.
        ├── deploy.yaml           # Complete DynamoGraphDeployment manifest
        └── perf.yaml (optional)  # AIPerf benchmark job
43
44
45
46
```

## Quick Start

47
### Prerequisites
48

49
**1. Dynamo Platform Installed**
50

51
The recipes require the Dynamo Kubernetes Platform to be installed. Follow the installation guide:
52

53
54
- **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** - Quickstart (~10 minutes)
- **[Detailed Installation Guide](../docs/kubernetes/installation_guide.md)** - Advanced options
55

56
**2. GPU Cluster Requirements**
57

58
59
Ensure your cluster has:
- GPU nodes matching recipe requirements (see table above)
60
- GPU operator installed
61
- Appropriate GPU drivers and container runtime
62

63
**3. HuggingFace Access**
64

65
Configure authentication to download models:
66
67

```bash
68
69
export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
70

71
72
73
74
# Create HuggingFace token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token-here" \
  -n ${NAMESPACE}
75
76
```

77
78
79
**4. Storage Configuration**

Update the `storageClassName` in `<model>/model-cache/model-cache.yaml` to match your cluster:
80
81

```bash
82
# Find your storage class name
83
kubectl get storageclass
84

85
86
87
# Edit the model-cache.yaml file and update:
# spec:
#   storageClassName: "your-actual-storage-class"
88
89
```

90
### Deploy a Recipe
91

92
**Step 1: Download Model**
93
94

```bash
95
cd recipes
96
97
# Update storageClassName in model-cache.yaml first!
kubectl apply -f <model>/model-cache/ -n ${NAMESPACE}
98

99
100
101
# Create model cache PVC
kubectl apply -f <model>/model-cache/model-download.yaml -n ${NAMESPACE}

102
103
# Wait for download to complete (may take 10-60 minutes depending on model size)
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s
104

105
106
107
# Monitor progress
kubectl logs -f job/model-download -n ${NAMESPACE}
```
108

109
**Step 2: Deploy Service**
110

111
112
Update the image in `<model>/<framework>/<mode>/deploy.yaml`.

113
```bash
114
kubectl apply -f <model>/<framework>/<mode>/deploy.yaml -n ${NAMESPACE}
115

116
117
# Check deployment status
kubectl get dynamographdeployment -n ${NAMESPACE}
118

119
120
# Check pod status
kubectl get pods -n ${NAMESPACE}
121

122
123
# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l nvidia.com/dynamo-graph-deployment-name=<deployment-name> -n ${NAMESPACE} --timeout=600s
124
```
125

126
**Step 3: Test Deployment**
127

128
129
130
```bash
# Port forward to access the service locally
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
131

132
133
# In another terminal, test the endpoint
curl http://localhost:8000/v1/models
134

135
136
137
138
139
140
141
142
# Send a test request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'
143
```
144

145
**Step 4: Run Benchmark (Optional)**
146
147

```bash
148
149
# Only if perf.yaml exists in the recipe directory
kubectl apply -f <model>/<framework>/<mode>/perf.yaml -n ${NAMESPACE}
150

151
152
# Monitor benchmark progress
kubectl logs -f job/<benchmark-job-name> -n ${NAMESPACE}
153

154
155
# View results after completion
kubectl logs job/<benchmark-job-name> -n ${NAMESPACE} | tail -50
156
157
158
```


159
## Example Deployments
160

161
### Llama-3-70B with vLLM (Aggregated)
162
163

```bash
164
165
export NAMESPACE=dynamo-demo
kubectl create namespace ${NAMESPACE}
166

167
168
169
170
# Create HF token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}
171

172
# Deploy
173
cd recipes
174
kubectl apply -f llama-3-70b/model-cache/ -n ${NAMESPACE}
175
kubectl apply -f llama-3-70b/model-cache/model-download.yaml -n ${NAMESPACE}
176
177
178
179
180
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s
kubectl apply -f llama-3-70b/vllm/agg/deploy.yaml -n ${NAMESPACE}

# Test
kubectl port-forward svc/llama3-70b-agg-frontend 8000:8000 -n ${NAMESPACE}
181
182
```

atchernych's avatar
atchernych committed
183
184
185
186
### Inference Gateway (GAIE) Integration (Optional)**

For Llama-3-70B with vLLM (Aggregated), an example of integration with the Inference Gateway is provided.

187
188
189
190
191
First, deploy the Dynamo Graph per instructions above.

Then follow [Deploy Inference Gateway Section 2](../deploy/inference-gateway/README.md#2-deploy-inference-gateway) to install GAIE.

Update the containers.epp.image in the deployment file, i.e. llama-3-70b/vllm/agg/gaie/k8s-manifests/epp/deployment.yaml. It should match the release tag and be in the format `nvcr.io/nvidia/ai-dynamo/frontend:<my-tag>` i.e. `nvcr.io/nvstaging/ai-dynamo/dynamo-frontend:0.7.0rc2-amd64`
192
193
194
195
196
The recipe assumes you are using Kubernetes discovery backend and sets the `DYN_DISCOVERY_BACKEND` env variable in the epp deployment. If you want to use etcd enable the lines below and remove the DYN_DISCOVERY_BACKEND env var.
```bash
- name: ETCD_ENDPOINTS
  value: "dynamo-platform-etcd.$(PLATFORM_NAMESPACE):2379" #  update dynamo-platform to appropriate namespace
```
atchernych's avatar
atchernych committed
197
198
199

```bash
export DEPLOY_PATH=llama-3-70b/vllm/agg/
200
# DEPLOY_PATH=<model>/<framework>/<mode>/
atchernych's avatar
atchernych committed
201
kubectl apply -R -f "$DEPLOY_PATH/gaie/k8s-manifests" -n "$NAMESPACE"
atchernych's avatar
atchernych committed
202
```
atchernych's avatar
atchernych committed
203

204
### DeepSeek-R1 on GB200 (Multi-node)
205

206
See [deepseek-r1/trtllm/disagg/wide_ep/gb200/deploy.yaml](deepseek-r1/trtllm/disagg/wide_ep/gb200/deploy.yaml) for the complete multi-node WideEP configuration.
207

208
## Customization
209

210
211
212
213
214
Each `deploy.yaml` contains:
- **ConfigMap**: Engine-specific configuration (embedded in the manifest)
- **DynamoGraphDeployment**: Kubernetes resource definitions
- **Resource limits**: GPU count, memory, CPU requests/limits
- **Image references**: Container images with version tags
215

216
### Key Customization Points
217

218
219
220
221
222
223
**Model Configuration:**
```yaml
# In deploy.yaml under worker args:
args:
  - python3 -m dynamo.vllm --model <your-model-path> --served-model-name <name>
```
224

225
226
227
228
229
230
231
232
**GPU Resources:**
```yaml
resources:
  limits:
    gpu: "4"  # Adjust based on your requirements
  requests:
    gpu: "4"
```
233

234
235
236
237
238
239
**Scaling:**
```yaml
services:
  VllmDecodeWorker:
    replicas: 2  # Scale to multiple workers
```
240

241
242
243
244
245
246
247
**Router Mode:**
```yaml
# In Frontend args:
args:
  - python3 -m dynamo.frontend --router-mode kv --http-port 8000
# Options: round-robin, kv (KV-aware routing)
```
248

249
250
251
252
**Container Images:**
```yaml
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:x.y.z
# Update version tag as needed
253
254
```

255
## Troubleshooting
256

257
### Common Issues
258

259
260
261
262
**Pods stuck in Pending:**
- Check GPU availability: `kubectl describe node <node-name>`
- Verify storage class exists: `kubectl get storageclass`
- Check resource requests vs. available resources
263

264
265
266
267
**Model download fails:**
- Verify HuggingFace token is correct
- Check network connectivity from cluster
- Review job logs: `kubectl logs job/model-download -n ${NAMESPACE}`
268

269
270
271
272
**Workers fail to start:**
- Check GPU compatibility (driver version, CUDA version)
- Verify image pull secrets if using private registries
- Review pod logs: `kubectl logs <pod-name> -n ${NAMESPACE}`
273

274
275
276
**For more troubleshooting:**
- [Kubernetes Deployment Guide](../docs/kubernetes/README.md#troubleshooting)
- [Observability Documentation](../docs/kubernetes/observability/)
277

278
## Related Documentation
279

280
281
282
283
284
285
286
- **[Kubernetes Deployment Guide](../docs/kubernetes/README.md)** - Platform installation and concepts
- **[API Reference](../docs/kubernetes/api_reference.md)** - DynamoGraphDeployment CRD specification
- **[vLLM Backend Guide](../docs/backends/vllm/README.md)** - vLLM-specific features
- **[SGLang Backend Guide](../docs/backends/sglang/README.md)** - SGLang-specific features
- **[TensorRT-LLM Backend Guide](../docs/backends/trtllm/README.md)** - TensorRT-LLM features
- **[Observability](../docs/kubernetes/observability/)** - Monitoring and logging
- **[Benchmarking Guide](../docs/benchmarks/benchmarking.md)** - Performance testing
287

288
## Contributing
289

290
291
292
293
294
We welcome contributions of new recipes! See [CONTRIBUTING.md](CONTRIBUTING.md) for:
- Recipe submission guidelines
- Required components checklist
- Testing and validation requirements
- Documentation standards
295

296
297
298
299
300
301
302
303
### Recipe Quality Standards

A production-ready recipe must include:
- ✅ Complete `deploy.yaml` with DynamoGraphDeployment
- ✅ Model cache PVC and download job
- ✅ Benchmark recipe (`perf.yaml`) for performance testing
- ✅ Verification on target hardware
- ✅ Documentation of GPU requirements