@@ -189,61 +189,18 @@ For comprehensive instructions on multinode serving, see the [multinode-examples
...
@@ -189,61 +189,18 @@ For comprehensive instructions on multinode serving, see the [multinode-examples
### Kubernetes Deployment
### Kubernetes Deployment
For Kubernetes deployment, YAML manifests are provided in the `deploy/` directory. These define DynamoGraphDeployment resources for various configurations:
For complete Kubernetes deployment instructions, configurations, and troubleshooting, see [TensorRT-LLM Kubernetes Deployment Guide](deploy/README.md)
-`agg.yaml` - Aggregated serving
-`agg_router.yaml` - Aggregated serving with KV routing
-`disagg.yaml` - Disaggregated serving
-`disagg_router.yaml` - Disaggregated serving with KV routing
#### Prerequisites
-**Dynamo Cloud**: Follow the [Quickstart Guide](../../../docs/guides/dynamo_deploy/quickstart.md) to deploy Dynamo Cloud first.
-**Container Images**: The deployment files currently require access to `nvcr.io/nvidian/nim-llm-dev/trtllm-runtime`. If you don't have access, build and push your own image:
```bash
./container/build.sh --framework tensorrtllm
# Tag and push to your container registry
# Update the image references in the YAML files
```
-**Port Forwarding**: After deployment, forward the frontend service to access the API:
This directory contains Kubernetes Custom Resource Definition (CRD) templates for deploying TensorRT-LLM inference graphs using the **DynamoGraphDeployment** resource.
## Available Deployment Patterns
### 1. **Aggregated Deployment** (`agg.yaml`)
Basic deployment pattern with frontend and a single worker.
**Architecture:**
-`Frontend`: OpenAI-compatible API server (with kv router mode disabled)
-`TRTLLMWorker`: Single worker handling both prefill and decode
1.**Dynamo Cloud Platform installed** - See [Quickstart Guide](../../../../docs/guides/dynamo_deploy/quickstart.md)
2.**Kubernetes cluster with GPU support**
3.**Container registry access** for TensorRT-LLM runtime images
4.**HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`)
### Container Images
The deployment files currently require access to `nvcr.io/nvidian/nim-llm-dev/trtllm-runtime`. If you don't have access, build and push your own image:
```bash
./container/build.sh --framework tensorrtllm
# Tag and push to your container registry
# Update the image references in the YAML files
```
**Note:** TensorRT-LLM uses git-lfs, which needs to be installed in advance:
To change `DYN_LOG` level, edit the yaml file by adding:
```yaml
...
spec:
envs:
-name:DYN_LOG
value:"debug"# or other log levels
...
```
### TensorRT-LLM Worker Configuration
TensorRT-LLM workers are configured through command-line arguments in the deployment YAML. Key configuration areas include:
-**Disaggregation Strategy**: Control request flow with `DISAGGREGATION_STRATEGY` environment variable
-**KV Cache Transfer**: Choose between UCX (default) or NIXL for disaggregated serving
-**Request Migration**: Enable graceful failure handling with `--migration-limit`
### Disaggregation Strategy
The disaggregation strategy controls how requests are distributed between prefill and decode workers:
-**`decode_first`** (default): Requests routed to decode worker first, then forwarded to prefill worker
-**`prefill_first`**: Requests routed directly to prefill worker (used with KV routing)
Set via environment variable:
```yaml
envs:
-name:DISAGGREGATION_STRATEGY
value:"prefill_first"
```
## Testing the Deployment
Send a test request to verify your deployment. See the [client section](../../../../components/backends/llm/README.md#client) for detailed instructions.
**Note:** For multi-node deployments, target the node running `python3 -m dynamo.frontend <args>`.
## Model Configuration
The deployment templates support various TensorRT-LLM models and configurations. You can customize model-specific arguments in the worker configuration sections of the YAML files.
### Multi-Token Prediction (MTP) Support
For models supporting Multi-Token Prediction (such as DeepSeek R1), special configuration is available. Note that MTP requires the experimental TensorRT-LLM commit:
-**Frontend health endpoint**: `http://<frontend-service>:8000/health`
-**Worker health endpoints**: `http://<worker-service>:9090/health`
-**Liveness probes**: Check process health every 5 seconds
-**Readiness probes**: Check service readiness with configurable delays
## KV Cache Transfer Methods
TensorRT-LLM supports two methods for KV cache transfer in disaggregated serving:
-**UCX** (default): Standard method for KV cache transfer
-**NIXL** (experimental): Alternative transfer method
For detailed configuration instructions, see the [KV cache transfer guide](../kv-cache-tranfer.md).
## Request Migration
You can enable [request migration](../../../../docs/architecture/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations:
```yaml
args:
-"python3"
-"-m"
-"dynamo.trtllm"
-"--migration-limit"
-"3"
```
## Benchmarking
To benchmark your deployment with GenAI-Perf, see this utility script: [perf.sh](../../../../benchmarks/llm/perf.sh)
Configure the `model` name and `host` based on your deployment.
@@ -152,73 +152,7 @@ Below we provide a selected list of advanced deployments. Please open up an issu
...
@@ -152,73 +152,7 @@ Below we provide a selected list of advanced deployments. Please open up an issu
### Kubernetes Deployment
### Kubernetes Deployment
For Kubernetes deployment, YAML manifests are provided in the `deploy/` directory. These define DynamoGraphDeployment resources for various configurations:
For complete Kubernetes deployment instructions, configurations, and troubleshooting, see [vLLM Kubernetes Deployment Guide](deploy/README.md)
-`agg.yaml` - Aggregated serving
-`agg_router.yaml` - Aggregated serving with KV routing
-`disagg.yaml` - Disaggregated serving
-`disagg_router.yaml` - Disaggregated serving with KV routing
-`disagg_planner.yaml` - Disaggregated serving with [SLA Planner](../../../docs/architecture/sla_planner.md). See [SLA Planner Deployment Guide](../../../docs/guides/dynamo_deploy/sla_planner_deployment.md) for more details.
#### Prerequisites
-**Dynamo Cloud**: Follow the [Quickstart Guide](../../../docs/guides/dynamo_deploy/quickstart.md) to deploy Dynamo Cloud first.
-**Container Images**: We have public images available on [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts). If you'd prefer to use your own registry, build and push your own image:
```bash
./container/build.sh --framework VLLM
# Tag and push to your container registry
# Update the image references in the YAML files
```
-**Pre-Deployment Profiling (if Using SLA Planner)**: Follow the [pre-deployment profiling guide](../../../docs/architecture/pre_deployment_profiling.md) to run pre-deployment profiling. The results will be saved to the `profiling-pvc` PVC and queried by the SLA Planner.
-**Port Forwarding**: After deployment, forward the frontend service to access the API:
Export the NAMESPACE you used in your Dynamo Cloud Installation.
```bash
cd dynamo
cd components/backends/vllm/deploy
kubectl apply -f disagg.yaml -n$NAMESPACE
```
To change `DYN_LOG` level, edit the yaml file by adding
```yaml
...
spec:
envs:
-name:DYN_LOG
value:"debug"# or other log levels
...
```
### Testing the Deployment
Send a test request to verify your deployment:
```bash
curl localhost:8080/v1/chat/completions \
-H"Content-Type: application/json"\
-d'{
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
This folder contains examples for the VLLM inference backend.
# vLLM Kubernetes Deployment Configurations
\ No newline at end of file
This directory contains Kubernetes Custom Resource Definition (CRD) templates for deploying vLLM inference graphs using the **DynamoGraphDeployment** resource.
## Available Deployment Patterns
### 1. **Aggregated Deployment** (`agg.yaml`)
Basic deployment pattern with frontend and a single decode worker.
**Architecture:**
-`Frontend`: OpenAI-compatible API server (with kv router mode disabled)
-`VLLMDecodeWorker`: Single worker handling both prefill and decode
All templates use the **DynamoGraphDeployment** CRD:
```yaml
apiVersion:nvidia.com/v1alpha1
kind:DynamoGraphDeployment
metadata:
name:<deployment-name>
spec:
services:
<ServiceName>:
# Service configuration
```
### Key Configuration Options
**Resource Management:**
```yaml
resources:
requests:
cpu:"10"
memory:"20Gi"
gpu:"1"
limits:
cpu:"10"
memory:"20Gi"
gpu:"1"
```
**Container Configuration:**
```yaml
extraPodSpec:
mainContainer:
image:my-registry/vllm-runtime:my-tag
workingDir:/workspace/components/backends/vllm
args:
-"python3"
-"-m"
-"dynamo.vllm"
# Model-specific arguments
```
## Prerequisites
Before using these templates, ensure you have:
1.**Dynamo Cloud Platform installed** - See [Quickstart Guide](../../../../docs/guides/dynamo_deploy/quickstart.md)
2.**Kubernetes cluster with GPU support**
3.**Container registry access** for vLLM runtime images
4.**HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`)
### Container Images
We have public images available on [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts). If you'd prefer to use your own registry, build and push your own image:
```bash
./container/build.sh --framework VLLM
# Tag and push to your container registry
# Update the image references in the YAML files
```
### Pre-Deployment Profiling (SLA Planner Only)
If using the SLA Planner deployment (`disagg_planner.yaml`), follow the [pre-deployment profiling guide](../../../../docs/architecture/pre_deployment_profiling.md) to run pre-deployment profiling. The results will be saved to the `profiling-pvc` PVC and queried by the SLA Planner.
## Usage
### 1. Choose Your Template
Select the deployment pattern that matches your requirements:
- Use `agg.yaml` for simple testing
- Use `agg_router.yaml` for production with load balancing
- Use `disagg.yaml` for maximum performance
- Use `disagg_router.yaml` for high-performance with KV cache routing
- Use `disagg_planner.yaml` for SLA-optimized performance
### 2. Customize Configuration
Edit the template to match your environment:
```yaml
# Update image registry and tag
image:your-registry/vllm-runtime:your-tag
# Configure your model
args:
-"--model"
-"your-org/your-model"
```
### 3. Deploy
Use the following command to deploy the deployment file.
First, create a secret for the HuggingFace token.
```bash
export HF_TOKEN=your_hf_token
kubectl create secret generic hf-token-secret \
--from-literal=HF_TOKEN=${HF_TOKEN}\
-n${NAMESPACE}
```
Then, deploy the model using the deployment file.
Export the NAMESPACE you used in your Dynamo Cloud Installation.
```bash
cd <dynamo-source-root>/components/backends/vllm/deploy
export DEPLOYMENT_FILE=agg.yaml
kubectl apply -f$DEPLOYMENT_FILE-n$NAMESPACE
```
### 4. Using Custom Dynamo Frameworks Image for vLLM
To use a custom dynamo frameworks image for vLLM, you can update the deployment file using yq:
To change `DYN_LOG` level, edit the yaml file by adding:
```yaml
...
spec:
envs:
-name:DYN_LOG
value:"debug"# or other log levels
...
```
### vLLM Worker Configuration
vLLM workers are configured through command-line arguments. Key parameters include:
-`--endpoint`: Dynamo endpoint in format `dyn://namespace.component.endpoint`
-`--model`: Model to serve (e.g., `Qwen/Qwen3-0.6B`)
-`--is-prefill-worker`: Enable prefill-only mode for disaggregated serving
-`--metrics-endpoint-port`: Port for publishing KV metrics to Dynamo
See the [vLLM CLI documentation](https://docs.vllm.ai/en/v0.9.2/configuration/serve_args.html?h=serve+arg) for the full list of configuration options.
## Testing the Deployment
Send a test request to verify your deployment:
```bash
curl localhost:8080/v1/chat/completions \
-H"Content-Type: application/json"\
-d'{
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
}
],
"stream": false,
"max_tokens": 30
}'
```
## Model Configuration
All templates use **Qwen/Qwen3-0.6B** as the default model, but you can use any vLLM-supported LLM model and configuration arguments.
## Monitoring and Health
-**Frontend health endpoint**: `http://<frontend-service>:8000/health`
-**Liveness probes**: Check process health regularly
-**KV metrics**: Published via metrics endpoint port
## Request Migration
You can enable [request migration](../../../../docs/architecture/request_migration.md) to handle worker failures gracefully by adding the migration limit argument to worker configurations:
@@ -70,7 +70,17 @@ kubectl get gateway inference-gateway -n my-model
...
@@ -70,7 +70,17 @@ kubectl get gateway inference-gateway -n my-model
# inference-gateway kgateway x.x.x.x True 1m
# inference-gateway kgateway x.x.x.x True 1m
```
```
3.**Install dynamo model and dynamo gaie helm chart**
3.**Deploy model**
Follow the steps in [model deployment](../../components/backends/vllm/deploy/README.md) to deploy `Qwen/Qwen3-0.6B` model in aggregate mode using [agg.yaml](../../components/backends/vllm/deploy/agg.yaml) in `my-model` kubernetes namespace.
Sample commands to deploy model:
```bash
cd <dynamo-source-root>/components/backends/vllm/deploy
kubectl apply -f agg.yaml -n my-model
```
4.**Install Dynamo GAIE helm chart**
The Inference Gateway is configured through the `inference-gateway-resources.yaml` file.
The Inference Gateway is configured through the `inference-gateway-resources.yaml` file.
