SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# TensorRT-LLM Benchmark Scripts for DeepSeek R1 model
This directory contains scripts for benchmarking TensorRT-LLM performance with Dynamo using SLURM job scheduler.
## ⚠️ DISCLAIMER
**These scripts are currently not QA'ed and are provided for demonstration purposes only.**
Please note that:
- These scripts have not undergone formal quality assurance testing
- These scripts were tested on GB200 systems. To run all configurations, you will need at least 16 nodes, with each node equipped with 4 GPUs.
- They are intended for demonstration and educational purposes
- Use at your own risk in production environments
- Always review and test scripts thoroughly before running in your specific environment
- In disaggregated mode, using `--exclusive` flag to launch worker processes can impact runtime performance. Hence, these scripts specify nodelist explicitly in srun call.
- We are actively working on refining the configuration sweeps.
## Scripts Overview
### Core Scripts
1.`submit_disagg.sh` - Main entry point for submitting benchmark jobs for disaggregated configurations. This includes WideEP optimization for DEP>=16.
2.`submit_agg.sh` - Main entry point for submitting benchmark jobs for aggregated configurations.
3.`post_process.py` - Scan the aiperf results to produce a json with entries to each config point.
4.`plot_performance_comparison.py` - Takes the json result file for disaggregated and/or aggregated configuration sweeps and plots a pareto line for better visualization.
For more finer grained details on how to launch TRTLLM backend workers with DeepSeek R1 on GB200 slurm, please refer [multinode-examples.md](../../../../docs/backends/trtllm/multinode/trtllm-multinode-examples.md). This guide shares similar assumption to the multinode examples guide.
## Usage
### Prerequisites
Before running the scripts, ensure you have:
1. Access to a SLURM cluster
2. Container image of Dynamo with TensorRT-LLM built using instructions from [here](https://github.com/ai-dynamo/dynamo/tree/main/docs/backends/trtllm/README.md#build-container).
3. Model files accessible on the cluster
4. Required environment variables set
### Setup
Within the login node of the cluster, set the following variables
```bash
# Set partition manually based on your slurm cluster's partition names
export SLURM_PARTITION=""
# Set account manually if this command doesn't work on your cluster
export SLURM_ACCOUNT="$(sacctmgr -nP show assoc where user=$(whoami)format=account)"
# NOTE: In general, Deepseek R1 is very large, so it is recommended to
# pre-download the model weights and save them in some shared location,
# NFS storage, HF_HOME, etc. and modify the `--model-path` below
# to reuse the pre-downloaded weights instead.
#
# On Blackwell systems (ex: GB200), it is recommended to use the FP4 weights:
# https://huggingface.co/nvidia/DeepSeek-R1-FP4
#
# On Hopper systems, FP4 isn't supported so you'll need to use the default weights:
# https://huggingface.co/deepseek-ai/DeepSeek-R1
export MODEL_PATH="<path_to_model_weights>"
# The name the model will be served/queried under, matching what's
# returned by the /v1/models endpoint.
#
# By default this is inferred from MODEL_PATH, but when using locally downloaded
# model weights, it can be nice to have explicit control over the name.
export SERVED_MODEL_NAME="nvidia/DeepSeek-R1-FP4"
```
## Launching benchmarking sweeps for different configurations
### Aggregated
```bash
# Queues the SLURM jobs for aggregated configurations for DeepSeek R1.
./submit_agg.sh
```
### Disaggregated (Includes WideEP) - MTP off
```bash
# Queues the SLURM jobs for disaggregated configurations for DeepSeek R1 without MTP
./submit_disagg.sh mtp=off all
```
### Disaggregated (Includes WideEP) - MTP on
```bash
# Queues the SLURM jobs for disaggregated configurations for DeepSeek R1 with MTP
./submit.sh mtp=on all
```
## Post-Processing Results
The above jobs use aiperf tool to benchmark each configuration point across different concurrency values. These get stored in `dynamo_disagg-bm-8150-1024/<config-setup>/aiperf_artifacts` and `dynamo_agg-bm-8150-1024/<config-setup>/aiperf_artifacts` for disaggregated and aggregated respectively.
After your benchmarking jobs have completed, you can use the `post_process.py` script to aggregate and summarize the results from the generated aiperf_artifacts.
This script will produce a scatter plot of all the configuration points with each concurrency on a Output Throughput per GPU vs Output Throughput per User. It will also include the roofline pareto line for both aggregated and disaggregated setups.
Refer to [Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://arxiv.org/html/2506.05508v1) to learn how to interpret these plots.
## Known Issues
- Some jobs may time out if aiperf requires more time to complete all concurrency levels.
- Workers may encounter out-of-memory (OOM) errors during inference, especially with larger configurations.
- Configurations affected by these issues will result in missing data points on the performance plot.
f"Tasks for a instance {instance} of {ctx_or_gen} instances use more node than expected. Nodes used: {instance_nodes}, number of nodes expected: {min_node}, max_tasks_per_node: {max_tasks_per_node}"
)
node=task_nodes[task_nodes_offset]
port=node_to_port[node]
node_to_port[node]+=1
task_nodes_offset+=tasks_needed
urls.append(f"{node}:{port}")
print(f"{ctx_or_gen} urls: {urls}")
returnurls,task_nodes_offset
defgen_config_file(
config_path:str,
decode_config_path:str,
instance_config_path:str,
model_path:str,
num_ctx_servers:int,
ctx_tp_size:int,
ctx_batch_size:int,
ctx_max_num_tokens:int,
ctx_max_seq_len:int,
ctx_free_gpu_memory_fraction:float,
ctx_enable_attention_dp:bool,
num_gen_servers:int,
gen_tp_size:int,
gen_batch_size:int,
gen_max_num_tokens:int,
gen_max_seq_len:int,
gen_enable_attention_dp:bool,
gen_gpu_memory_fraction:float,
eplb_num_slots:int,
mtp_size:int=0,
worker_start_port:int=8001,
server_port:int=8000,
cache_transceiver_max_num_tokens:int=4608,
)->None:
"""
Generate configuration YAML file for disaggregated inference.
Args:
config_path: Path to save the config file
model_path: Path to the model
num_ctx_servers: Number of context servers
ctx_tp_size: Tensor parallel size for context servers
ctx_batch_size: Batch size for context servers
ctx_max_num_tokens: Max number of tokens for context servers
ctx_max_seq_len: Max sequence length for context servers
ctx_free_gpu_memory_fraction: Free GPU memory fraction for context servers
ctx_enable_attention_dp: Enable attention DP for context servers
num_gen_servers: Number of generation servers
gen_tp_size: Tensor parallel size for generation servers
gen_batch_size: Batch size for generation servers
gen_max_num_tokens: Max number of tokens for generation servers
gen_enable_attention_dp: Enable attention DP for generation servers
gen_gpu_memory_fraction: GPU memory fraction for generation servers
This example demonstrates Dynamo's **Prefill/Decode Disaggregated Serving** architecture, where the prefill and decode phases of LLM inference are separated into specialized workers for enhanced performance, improved resource utilization, and better scalability.
## What is P/D Disaggregated Serving?
Traditional LLM inference combines two distinct phases with different computational characteristics:
-**Prefill Phase**: Processes the entire input prompt to generate the KV cache (compute-bound)
-**Decode Phase**: Generates output tokens one by one using the KV cache (memory-bound)
Dynamo's disaggregated architecture separates these phases into specialized workers:
-**Prefill Workers**: Optimized for high-throughput parallel processing of input tokens
-**Decode Workers**: Optimized for low-latency sequential token generation
This separation allows for:
-**Better Hardware Utilization**: Use different parallelism configurations optimized for each phase
-**Improved Scalability**: Scale prefill and decode workers independently based on workload
-**Enhanced Performance**: Eliminate head-of-line blocking where long prefills delay ongoing decodes
## Prerequisites
> [!NOTE]
> This example requires having at least 2 GPUs -- one for Prefill and one for Decode
Before running this example, ensure you have the following services running:
-**etcd**: A distributed key-value store used for service discovery and metadata storage
-**NATS**: A high-performance message broker for inter-component communication
You can start these services using Docker Compose:
```bash
docker compose -f deploy/docker-compose.yml up -d
```
## Components
-[Frontend](/components/src/dynamo/frontend/README.md) - HTTP API endpoint that receives requests and forwards them to the decode worker
-[vLLM Prefill Worker](/docs/backends/vllm/README.md) - Specialized worker for prefill phase execution
-[vLLM Decode Worker](/docs/backends/vllm/README.md) - Specialized worker that handles requests and decides between local/remote prefill
```mermaid
---
title: Disaggregated Request Flow
---
flowchart TD
Client["Users/Clients<br/>(HTTP)"] --> Frontend["Frontend<br/>HTTP API endpoint<br/>(OpenAI Style)"]
- **Automatic Fallback**: When no prefill workers are available, decode workers handle everything locally
- **Transparent Operation**: Clients are unaware of whether requests are processed locally or disaggregated
This approach ensures the system remains operational regardless of configuration changes, automatically adapting to the available resources.
### 3. High-Performance KV Cache Transfer
The architecture relies on NVIDIA's NIXL (NVIDIA Inference Transfer Library) for efficient KV cache movement:
- **Direct GPU-to-GPU Transfer**: KV cache data moves directly between GPU memory without CPU involvement
- **Zero-Copy Operations**: Eliminates redundant memory copies for maximum efficiency
- **Automatic Transport Selection**: NIXL chooses the optimal transport (NVLink, InfiniBand, etc.) based on hardware topology
### 4. Request Flow
```mermaid
sequenceDiagram
participant Client
participant Decode as Decode Worker
participant Prefill as Prefill Worker
Client->>Decode: Send request
Decode->>Decode: Check prefill availability
alt Prefill workers available
Decode->>Prefill: Forward for prefill
Prefill->>Prefill: Compute KV cache
Note over Prefill,Decode: NIXL transfers KV cache
Prefill-->>Decode: Return control
Decode->>Decode: Generate tokens
else No prefill workers
Decode->>Decode: Prefill + Decode locally
end
Decode-->>Client: Stream response tokens
```
### 5. Key Benefits
This disaggregated architecture provides several advantages:
1. **Resource Optimization**: Each worker type can be optimized for its specific workload
2. **Independent Scaling**: Add prefill or decode workers based on workload characteristics
3. **Improved Latency**: Ongoing decode operations aren't blocked by new prefill requests
4. **Seamless Degradation**: System continues operating even without prefill workers
### 6. Operational Flexibility
The architecture supports various deployment patterns:
- **Single Node**: Prefill and decode workers on different GPUs of the same machine
- **Multi-Node**: Workers distributed across multiple machines for larger scale
- **Dynamic Scaling**: Add or remove workers without disrupting ongoing operations
By separating concerns and using efficient communication mechanisms, Dynamo achieves the performance benefits of disaggregation without the complexity typically associated with distributed systems.
## 1. Single-Node-Sized Models hosting on multiple Nodes
For SNS (Single-Node-Sized) Model, we can use Dynamo aggregated serving to deploy multiple replicas of the model and create a frontend with different routing strategies
1. Install Dynamo CRD
```sh
export RELEASE_VERSION=0.5.0 # any version of Dynamo 0.3.2+
This `agg_router.yaml` is adpated from vLLM deployment [example](https://github.com/ai-dynamo/dynamo/blob/main/examples/backends/vllm/deploy/agg_router.yaml). It has following customizations
- Deployed `Qwen/Qwen2.5-1.5B-Instruct` model
- Use KV cache based routing in frontend deployment via the `DYN_ROUTER_MODE=kv` environment variable
- Mounted a local cache folder `/YOUR/LOCAL/CACHE/FOLDER` for model artifacts reuse
- Created 4 replicas for this model deployment by setting `replicas: 4`
- Added `debug` flag environment variable for observability
Create a K8S secret with your Huggingface token and then deploy the models
and use following request to test the deployed model
```sh
curl localhost:8000/v1/chat/completions \
-H"Content-Type: application/json"\
-d'{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [
{
"role": "user",
"content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
}
],
"stream": false,
"max_tokens": 30
}'
```
You can also benchmark the performance of the endpoint by [AIPerf](https://github.com/ai-dynamo/aiperf/blob/main/README.md)
## 2. Deploy Single-Node-Sized Models using AIConfigurator
AIConfigurator helps users to find a strong starting configuration for disaggregated serving. We can use it as a guidance for the SNS (Single-Node-Sized) Model's serving.
1. Install AI Configurator
```sh
pip3 install aiconfigurator
```
2. Assume we have 2 GPU nodes with 16 H200 in total, and we want to deploy Llama 3.1-70B-Instruct model with an optimal disaggregated serving configuration. Run AI configurator for this model
and from the output, you can see the Pareto curve with the suggested P/D settings
3. Start the serving with 1 prefill worker with tensor parallelism 4 and 1 decoding worker with tensor parallelism 8 as AI Configurator suggested. Update the `my-tag` in `disagg_router.yaml` with the latest Dynamo version and your local cache folder path and run following command.
