This example demonstrates Dynamo's **Prefill/Decode Disaggregated Serving** architecture, where the prefill and decode phases of LLM inference are separated into specialized workers for enhanced performance, improved resource utilization, and better scalability.
## What is P/D Disaggregated Serving?
Traditional LLM inference combines two distinct phases with different computational characteristics:
-**Prefill Phase**: Processes the entire input prompt to generate the KV cache (compute-bound)
-**Decode Phase**: Generates output tokens one by one using the KV cache (memory-bound)
Dynamo's disaggregated architecture separates these phases into specialized workers:
-**Prefill Workers**: Optimized for high-throughput parallel processing of input tokens
-**Decode Workers**: Optimized for low-latency sequential token generation
This separation allows for:
-**Better Hardware Utilization**: Use different parallelism configurations optimized for each phase
-**Improved Scalability**: Scale prefill and decode workers independently based on workload
-**Enhanced Performance**: Eliminate head-of-line blocking where long prefills delay ongoing decodes
## Prerequisites
> [!NOTE]
> This example requires having at least 2 GPUs -- one for Prefill and one for Decode
Before running this example, ensure you have the following services running:
-**etcd**: A distributed key-value store used for service discovery and metadata storage
-**NATS**: A high-performance message broker for inter-component communication
You can start these services using Docker Compose:
```bash
docker compose -f deploy/metrics/docker-compose.yml up -d
```
## Components
-[Frontend](../../../components/frontend/README) - HTTP API endpoint that receives requests and forwards them to the decode worker
-[vLLM Prefill Worker](../../../components/backends/vllm/README) - Specialized worker for prefill phase execution
-[vLLM Decode Worker](../../../components/backends/vllm/README) - Specialized worker that handles requests and decides between local/remote prefill
```mermaid
---
title: Disaggregated Request Flow
---
flowchart TD
Client["Users/Clients<br/>(HTTP)"] --> Frontend["Frontend<br/>HTTP API endpoint<br/>(OpenAI Style)"]
- **Automatic Fallback**: When no prefill workers are available, decode workers handle everything locally
- **Transparent Operation**: Clients are unaware of whether requests are processed locally or disaggregated
This approach ensures the system remains operational regardless of configuration changes, automatically adapting to the available resources.
### 3. High-Performance KV Cache Transfer
The architecture relies on NVIDIA's NIXL (NVIDIA Inference Transfer Library) for efficient KV cache movement:
- **Direct GPU-to-GPU Transfer**: KV cache data moves directly between GPU memory without CPU involvement
- **Zero-Copy Operations**: Eliminates redundant memory copies for maximum efficiency
- **Automatic Transport Selection**: NIXL chooses the optimal transport (NVLink, InfiniBand, etc.) based on hardware topology
### 4. Request Flow
```mermaid
sequenceDiagram
participant Client
participant Decode as Decode Worker
participant Prefill as Prefill Worker
Client->>Decode: Send request
Decode->>Decode: Check prefill availability
alt Prefill workers available
Decode->>Prefill: Forward for prefill
Prefill->>Prefill: Compute KV cache
Note over Prefill,Decode: NIXL transfers KV cache
Prefill-->>Decode: Return control
Decode->>Decode: Generate tokens
else No prefill workers
Decode->>Decode: Prefill + Decode locally
end
Decode-->>Client: Stream response tokens
```
### 5. Key Benefits
This disaggregated architecture provides several advantages:
1. **Resource Optimization**: Each worker type can be optimized for its specific workload
2. **Independent Scaling**: Add prefill or decode workers based on workload characteristics
3. **Improved Latency**: Ongoing decode operations aren't blocked by new prefill requests
4. **Seamless Degradation**: System continues operating even without prefill workers
### 6. Operational Flexibility
The architecture supports various deployment patterns:
- **Single Node**: Prefill and decode workers on different GPUs of the same machine
- **Multi-Node**: Workers distributed across multiple machines for larger scale
- **Dynamic Scaling**: Add or remove workers without disrupting ongoing operations
By separating concerns and using efficient communication mechanisms, Dynamo achieves the performance benefits of disaggregation without the complexity typically associated with distributed systems.
