docs: migrate existing docs to fern (#5445)

Signed-off-by: Jont828 <jt572@cornell.edu>
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
Co-authored-by: Neal Vaidya <nealv@nvidia.com>

fern/pages/kvbm/kvbm-motivation.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Motivation behind KVBM"
+---
+
+Large language models (LLMs) and other AI workloads increasingly rely on KV caches that extend beyond GPU and local CPU memory into remote storage tiers. However, efficiently managing the lifecycle of KV blocks in remote storage presents challenges:
+
+* Tailored for GenAI use-cases
+* Lack of visibility into real-time block usage patterns.
+* Need for lightweight, ownership-driven memory management over complex object stores with unneeded overheads.
+* Modular and need simplified UX and to be memory safe.
+* Inability to differentiate between hot (frequently accessed) and cold (infrequently accessed) blocks across the stack without intrusive application-level changes.
+* Difficulty in optimizing storage placement across heterogeneous storage tiers (for example, SSDs, object storage, and cloud storage).
+
+Conventional systems either lack dynamic feedback mechanisms or require deep integration into core storage paths, which both increases complexity and reduces portability.
+
+## Benefits of KV Cache offloading
+
+KV Cache offloading avoids expensive KV Cache recomputation, resulting in faster response times and a better user experience. In the end, providers benefit from higher throughput and lower cost per token, making their inference services more scalable and efficient.
+
+## When to offload KV Cache for reuse
+
+Offloading KV cache to CPU or storage is most effective when KV Cache exceeds GPU memory and cache reuse outweighs the overhead of transferring data. It is especially valuable in long-context, high-concurrency, or resource-constrained inference environments such as:
+
+* **Long sessions and multi-turn conversations:** Offloading preserves large prompt prefixes, avoids recomputation, and improves first-token latency and throughput.
+* **High concurrency (future):** Idle or partial conversations can be moved out of GPU memory, allowing active requests to proceed without hitting memory limits.
+* **Shared or repeated content (future):** Reuse across users or sessions (for example, system prompts and templates) increases cache hits, especially with remote or cross-instance sharing.
+* **Memory- or cost-constrained deployments:** Offloading to RAM or SSD reduces GPU demand, allowing longer prompts or more users without adding hardware.

fern/pages/kvbm/kvbm-reading.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "KVBM Further Reading"
+---
+
+- [vLLM](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html)
+- [SGLang](https://github.com/sgl-project/sglang/tree/main/benchmark/hicache)
+- [EMOGI](https://arxiv.org/abs/2006.06890)
\ No newline at end of file

fern/pages/kvbm/trtllm-setup.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Running KVBM in TensorRT-LLM"
+---
+
+This guide explains how to leverage KVBM (KV Block Manager) to manage KV cache and do KV offloading in TensorRT-LLM (trtllm).
+
+To learn what KVBM is, please check [here](kvbm-architecture.md)
+
+<Note>
+- Ensure that `etcd` and `nats` are running before starting.
+- KVBM only supports TensorRT-LLM’s PyTorch backend.
+- Disable partial reuse `enable_partial_reuse: false` in the LLM API config’s `kv_connector_config` to increase offloading cache hits.
+- KVBM requires TensorRT-LLM v1.1.0rc5 or newer.
+- Enabling KVBM metrics with TensorRT-LLM is still a work in progress.
+</Note>
+
+## Quick Start
+
+To use KVBM in TensorRT-LLM, you can follow the steps below:
+
+```bash
+# Start up etcd for KVBM leader/worker registration and discovery
+docker compose -f deploy/docker-compose.yml up -d
+
+# Build a dynamo TRTLLM container (KVBM is built in by default)
+./container/build.sh --framework trtllm
+
+# Launch the container
+./container/run.sh --framework trtllm -it --mount-workspace --use-nixl-gds
+
+# Configure KVBM cache tiers (choose one of the following options):
+
+# Option 1: CPU cache only (GPU -> CPU offloading)
+# 4 means 4GB of pinned CPU memory would be used
+export DYN_KVBM_CPU_CACHE_GB=4
+
+# Option 2: Both CPU and Disk cache (GPU -> CPU -> Disk tiered offloading)
+export DYN_KVBM_CPU_CACHE_GB=4
+# 8 means 8GB of disk would be used
+export DYN_KVBM_DISK_CACHE_GB=8
+
+# [Experimental] Option 3: Disk cache only (GPU -> Disk direct offloading, bypassing CPU)
+# NOTE: this option is only experimental and it might not give out the best performance.
+# NOTE: disk offload filtering is not supported when using this option.
+export DYN_KVBM_DISK_CACHE_GB=8
+
+# Note: You can also use DYN_KVBM_CPU_CACHE_OVERRIDE_NUM_BLOCKS or
+# DYN_KVBM_DISK_CACHE_OVERRIDE_NUM_BLOCKS to specify exact block counts instead of GB
+```
+
+<Note>
+When disk offloading is enabled, to extend SSD lifespan, disk offload filtering would be enabled by default. The current policy is only offloading KV blocks from CPU to disk if the blocks have frequency equal or more than `2`. Frequency is determined via doubling on cache hit (init with 1) and decrement by 1 on each time decay step.
+To disable disk offload filtering, set `DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER` to true or 1.
+</Note>
+
+```bash
+# write an example LLM API config
+# Note: Disable partial reuse "enable_partial_reuse: false" in the LLM API config’s "kv_connector_config" to increase offloading cache hits.
+cat > "/tmp/kvbm_llm_api_config.yaml" <<EOF
+backend: pytorch
+cuda_graph_config: null
+kv_cache_config:
+  enable_partial_reuse: false
+  free_gpu_memory_fraction: 0.80
+kv_connector_config:
+  connector_module: kvbm.trtllm_integration.connector
+  connector_scheduler_class: DynamoKVBMConnectorLeader
+  connector_worker_class: DynamoKVBMConnectorWorker
+EOF
+
+# [DYNAMO] start dynamo frontend
+python3 -m dynamo.frontend --http-port 8000 &
+
+# [DYNAMO] To serve an LLM model with dynamo
+python3 -m dynamo.trtllm \
+  --model-path Qwen/Qwen3-0.6B \
+  --served-model-name Qwen/Qwen3-0.6B \
+  --extra-engine-args /tmp/kvbm_llm_api_config.yaml &
+
+# Make a call to LLM
+curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
+    "model": "Qwen/Qwen3-0.6B",
+    "messages": [
+    {
+        "role": "user",
+        "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
+    }
+    ],
+    "stream":false,
+    "max_tokens": 30
+  }'
+
+```
+
+Alternatively, can use "trtllm-serve" with KVBM by replacing the above two [DYNAMO] cmds with below:
+```bash
+trtllm-serve Qwen/Qwen3-0.6B --host localhost --port 8000 --backend pytorch --extra_llm_api_options /tmp/kvbm_llm_api_config.yaml
+```
+
+## Enable and View KVBM Metrics
+
+Follow below steps to enable metrics collection and view via Grafana dashboard:
+```bash
+# Start the basic services (etcd & natsd), along with Prometheus and Grafana
+docker compose -f deploy/docker-observability.yml up -d
+
+# Set env var DYN_KVBM_METRICS to true, when launch via dynamo
+# Optionally set DYN_KVBM_METRICS_PORT to choose the /metrics port (default: 6880).
+DYN_KVBM_METRICS=true \
+DYN_KVBM_CPU_CACHE_GB=20 \
+python3 -m dynamo.trtllm \
+  --model-path Qwen/Qwen3-0.6B \
+  --served-model-name Qwen/Qwen3-0.6B \
+  --extra-engine-args /tmp/kvbm_llm_api_config.yaml &
+
+# Optional if firewall blocks KVBM metrics ports to send prometheus metrics
+sudo ufw allow 6880/tcp
+```
+
+View grafana metrics via http://localhost:3000 (default login: dynamo/dynamo) and look for KVBM Dashboard
+
+KVBM currently provides following types of metrics out of the box:
+- `kvbm_matched_tokens`: The number of matched tokens
+- `kvbm_offload_blocks_d2h`: The number of offload blocks from device to host
+- `kvbm_offload_blocks_h2d`: The number of offload blocks from host to disk
+- `kvbm_offload_blocks_d2d`: The number of offload blocks from device to disk (bypassing host memory)
+- `kvbm_onboard_blocks_d2d`: The number of onboard blocks from disk to device
+- `kvbm_onboard_blocks_h2d`: The number of onboard blocks from host to device
+- `kvbm_host_cache_hit_rate`: Host cache hit rate (0.0-1.0) from sliding window
+- `kvbm_disk_cache_hit_rate`: Disk cache hit rate (0.0-1.0) from sliding window
+
+## Troubleshooting
+
+1. If enabling KVBM does not show any TTFT perf gain or even perf degradation, one potential reason is not enough prefix cache hit on KVBM to reuse offloaded KV blocks.
+To confirm, please enable KVBM metrics as mentioned above and check the grafana dashboard `Onboard Blocks - Host to Device` and `Onboard Blocks - Disk to Device`.
+If observed large number of onboarded KV blocks as the example below, we can rule out this cause:
+![Grafana Example](../../assets/img/kvbm-metrics-grafana.png)
+
+2. Allocating large memory and disk storage can take some time and lead to KVBM worker initialization timeout.
+To avoid it, please set a longer timeout (default 1800 seconds) for leader–worker initialization.
+
+```bash
+# 3600 means 3600 seconds timeout
+export DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS=3600
+```
+
+3. When offloading to disk is enabled, KVBM could fail to start up if fallocate is not supported to create the files.
+To bypass the issue, please use disk zerofill fallback.
+
+```bash
+# Set to true to enable fallback behavior when disk operations fail (e.g. fallocate not available)
+export DYN_KVBM_DISK_ZEROFILL_FALLBACK=true
+```
+
+## Benchmark KVBM
+
+Once the model is loaded ready, follow below steps to use LMBenchmark to benchmark KVBM performance:
+```bash
+git clone https://github.com/LMCache/LMBenchmark.git
+
+# Show case of running the synthetic multi-turn chat dataset.
+# We are passing model, endpoint, output file prefix and qps to the sh script.
+cd LMBenchmark/synthetic-multi-round-qa
+./long_input_short_output_run.sh \
+    "Qwen/Qwen3-0.6B" \
+    "http://localhost:8000" \
+    "benchmark_kvbm" \
+    1
+
+# Average TTFT and other perf numbers would be in the output from above cmd
+```
+More details about how to use LMBenchmark could be found [here](https://github.com/LMCache/LMBenchmark).
+
+`NOTE`: if metrics are enabled as mentioned in the above section, you can observe KV offloading, and KV onboarding in the grafana dashboard.
+
+To compare, you can remove the `kv_connector_config` section from the LLM API config and run `trtllm-serve` with the updated config as the baseline.
+```bash
+cat > "/tmp/llm_api_config.yaml" <<EOF
+backend: pytorch
+cuda_graph_config: null
+kv_cache_config:
+  enable_partial_reuse: false
+  free_gpu_memory_fraction: 0.80
+EOF
+
+# Run trtllm-serve for the baseline for comparison
+trtllm-serve Qwen/Qwen3-0.6B --host localhost --port 8000 --backend pytorch --extra_llm_api_options /tmp/llm_api_config.yaml &
+```
+
+## Developing Locally
+
+Inside the Dynamo container, after changing KVBM related code (Rust and/or Python), to test or use it:
+```bash
+cd /workspace/lib/bindings/kvbm
+uv pip install maturin[patchelf]
+maturin build --release --out /workspace/dist
+uv pip install --upgrade --force-reinstall --no-deps /workspace/dist/kvbm*.whl
+```

fern/pages/kvbm/vllm-setup.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Running KVBM in vLLM"
+---
+
+This guide explains how to leverage KVBM (KV Block Manager) to manage KV cache and do KV offloading in vLLM.
+
+To learn what KVBM is, please check [here](kvbm-architecture.md)
+
+## Quick Start
+
+To use KVBM in vLLM, you can follow the steps below:
+
+### Docker Setup
+```bash
+# Start up etcd for KVBM leader/worker registration and discovery
+docker compose -f deploy/docker-compose.yml up -d
+
+# Build a dynamo vLLM container (KVBM is built in by default)
+./container/build.sh --framework vllm
+
+# Launch the container
+./container/run.sh --framework vllm -it --mount-workspace --use-nixl-gds
+```
+
+### Aggregated Serving with KVBM
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+./launch/agg_kvbm.sh
+```
+
+### Disaggregated Serving with KVBM
+```bash
+# 1P1D - one prefill worker and one decode worker
+# NOTE: need at least 2 GPUs
+cd $DYNAMO_HOME/examples/backends/vllm
+./launch/disagg_kvbm.sh
+
+# 2P2D - two prefill workers and two decode workers
+# NOTE: need at least 4 GPUs
+cd $DYNAMO_HOME/examples/backends/vllm
+./launch/disagg_kvbm_2p2d.sh
+```
+
+<Note>
+Configure or tune KVBM cache tiers (choose one of the following options):
+```bash
+# Option 1: CPU cache only (GPU -> CPU offloading)
+# 4 means 4GB of pinned CPU memory would be used
+export DYN_KVBM_CPU_CACHE_GB=4
+# Option 2: Both CPU and Disk cache (GPU -> CPU -> Disk tiered offloading)
+export DYN_KVBM_CPU_CACHE_GB=4
+# 8 means 8GB of disk would be used
+export DYN_KVBM_DISK_CACHE_GB=8
+# [Experimental] Option 3: Disk cache only (GPU -> Disk direct offloading, bypassing CPU)
+# NOTE: this option is only experimental and it might not give out the best performance.
+# NOTE: disk offload filtering is not supported when using this option.
+export DYN_KVBM_DISK_CACHE_GB=8
+```
+You can also use "DYN_KVBM_CPU_CACHE_OVERRIDE_NUM_BLOCKS" or
+"DYN_KVBM_DISK_CACHE_OVERRIDE_NUM_BLOCKS" to specify exact block counts instead of GB
+</Note>
+
+<Note>
+When disk offloading is enabled, to extend SSD lifespan, disk offload filtering would be enabled by default. The current policy is only offloading KV blocks from CPU to disk if the blocks have frequency equal or more than `2`. Frequency is determined via doubling on cache hit (init with 1) and decrement by 1 on each time decay step.
+To disable disk offload filtering, set `DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER` to true or 1.
+</Note>
+
+### Sample Request
+```bash
+# Make a request to verify vLLM with KVBM is started up correctly
+# NOTE: change the model name if served with a different one
+curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
+    "model": "Qwen/Qwen3-0.6B",
+    "messages": [
+    {
+        "role": "user",
+        "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
+    }
+    ],
+    "stream":false,
+    "max_tokens": 10
+  }'
+```
+
+Alternatively, can use `vllm serve` directly to use KVBM for aggregated serving:
+```bash
+vllm serve --kv-transfer-config '{"kv_connector":"DynamoConnector","kv_role":"kv_both", "kv_connector_module_path": "kvbm.vllm_integration.connector"}' Qwen/Qwen3-0.6B
+```
+
+## Enable and View KVBM Metrics
+
+Follow below steps to enable metrics collection and view via Grafana dashboard:
+```bash
+# Start the basic services (etcd & natsd), along with Prometheus and Grafana
+docker compose -f deploy/docker-observability.yml up -d
+
+# Set env var DYN_KVBM_METRICS to true, when launch via dynamo
+# Optionally set DYN_KVBM_METRICS_PORT to choose the /metrics port (default: 6880).
+# NOTE: update launch/disagg_kvbm.sh or launch/disagg_kvbm_2p2d.sh as needed
+DYN_KVBM_METRICS=true \
+DYN_KVBM_CPU_CACHE_GB=20 \
+python -m dynamo.vllm \
+    --model Qwen/Qwen3-0.6B \
+    --enforce-eager \
+    --connector kvbm
+
+# Optional, if firewall blocks KVBM metrics ports to send prometheus metrics
+sudo ufw allow 6880/tcp
+```
+
+View grafana metrics via http://localhost:3000 (default login: dynamo/dynamo) and look for KVBM Dashboard
+
+KVBM currently provides following types of metrics out of the box:
+- `kvbm_matched_tokens`: The number of matched tokens
+- `kvbm_offload_blocks_d2h`: The number of offload blocks from device to host
+- `kvbm_offload_blocks_h2d`: The number of offload blocks from host to disk
+- `kvbm_offload_blocks_d2d`: The number of offload blocks from device to disk (bypassing host memory)
+- `kvbm_onboard_blocks_d2d`: The number of onboard blocks from disk to device
+- `kvbm_onboard_blocks_h2d`: The number of onboard blocks from host to device
+- `kvbm_host_cache_hit_rate`: Host cache hit rate (0.0-1.0) from sliding window
+- `kvbm_disk_cache_hit_rate`: Disk cache hit rate (0.0-1.0) from sliding window
+
+## Troubleshooting
+
+1. If enabling KVBM does not show any TTFT perf gain or even perf degradation, one potential reason is not enough prefix cache hit on KVBM to reuse offloaded KV blocks.
+To confirm, please enable KVBM metrics as mentioned above and check the grafana dashboard `Onboard Blocks - Host to Device` and `Onboard Blocks - Disk to Device`.
+If observed large number of onboarded KV blocks as the example below, we can rule out this cause:
+![Grafana Example](../../assets/img/kvbm-metrics-grafana.png)
+
+2. Allocating large memory and disk storage can take some time and lead to KVBM worker initialization timeout.
+To avoid it, please set a longer timeout (default 1800 seconds) for leader–worker initialization.
+
+```bash
+# 3600 means 3600 seconds timeout
+export DYN_KVBM_LEADER_WORKER_INIT_TIMEOUT_SECS=3600
+```
+
+3. When offloading to disk is enabled, KVBM could fail to start up if fallocate is not supported to create the files.
+To bypass the issue, please use disk zerofill fallback.
+
+```bash
+# Set to true to enable fallback behavior when disk operations fail (e.g. fallocate not available)
+export DYN_KVBM_DISK_ZEROFILL_FALLBACK=true
+```
+
+## Benchmark KVBM
+
+Once the model is loaded ready, follow below steps to use LMBenchmark to benchmark KVBM performance:
+```bash
+git clone https://github.com/LMCache/LMBenchmark.git
+
+# Show case of running the synthetic multi-turn chat dataset.
+# We are passing model, endpoint, output file prefix and qps to the sh script.
+cd LMBenchmark/synthetic-multi-round-qa
+./long_input_short_output_run.sh \
+    "Qwen/Qwen3-0.6B" \
+    "http://localhost:8000" \
+    "benchmark_kvbm" \
+    1
+
+# Average TTFT and other perf numbers would be in the output from above cmd
+```
+More details about how to use LMBenchmark could be found [here](https://github.com/LMCache/LMBenchmark).
+
+`NOTE`: if metrics are enabled as mentioned in the above section, you can observe KV offloading, and KV onboarding in the grafana dashboard.
+
+To compare, you can run `vllm serve Qwen/Qwen3-0.6B` to turn KVBM off as the baseline.
+
+## Developing Locally
+
+Inside the Dynamo container, after changing KVBM related code (Rust and/or Python), to test or use it:
+```bash
+cd /workspace/lib/bindings/kvbm
+uv pip install maturin[patchelf]
+maturin build --release --out /workspace/dist
+uv pip install --upgrade --force-reinstall --no-deps /workspace/dist/kvbm*.whl
+```

fern/pages/multimodal/index.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Multimodal Inference in Dynamo"
+---
+
+Dynamo supports multimodal inference across multiple LLM backends, enabling models to process images, video, and audio alongside text. This section provides comprehensive documentation for deploying multimodal models.
+
+<Warning>
+**Security Requirement**: Multimodal processing must be explicitly enabled at startup.
+See the relevant documentation for each backend for the necessary flags.
+This prevents unintended processing of multimodal data from untrusted sources.
+</Warning>
+
+## Backend Documentation
+
+
+
+## Support Matrix
+
+### Backend Capabilities
+
+| Stack | E/PD | E/P/D | EP/D | EPD | Image | Video | Audio |
+|-------|------|-------|------|-----|-------|-------|-------|
+| **[vLLM](vllm.md)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🧪 |
+| **[TRT-LLM](trtllm.md)** | ❌ | 🚧* | ✅ | ✅ | ✅ | ❌ | ❌ |
+| **[SGLang](sglang.md)** | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
+
+\* E/P/D supported in TRT-LLM with pre-computed embeddings only; image URL support is WIP ([PR #4668](https://github.com/ai-dynamo/dynamo/pull/4668))
+
+**Pattern Key:**
+
+- **EPD** - All-in-one worker (Simple Aggregated)
+- **E/PD** - Separate encode, combined prefill+decode
+- **E/P/D** - All stages separate
+- **EP/D** - Combined encode+prefill, separate decode
+
+**Status:** ✅ Supported | 🚧 WIP | 🧪 Experimental | ❌ Not supported
+
+### Input Format Support
+
+| Format | vLLM | TRT-LLM | SGLang |
+|--------|------|---------|--------|
+| HTTP/HTTPS URL | ✅ | ✅ | ✅ |
+| Data URL (Base64) | ✅ | ❌ | ❌ |
+| Pre-computed Embeddings (.pt) | ❌ | ✅ | ❌ |
+
+## Architecture Patterns
+
+Dynamo supports several deployment patterns for multimodal inference based on two dimensions:
+
+1. **Encoding**: Is media encoding handled inline (within prefill) or by a separate **Encode Worker**?
+   - *Inline*: Simpler setup, encoding happens in the prefill worker
+   - *Separate (EPD)*: Dedicated encode worker transfers embeddings via **NIXL (RDMA)**, enabling independent scaling
+
+2. **Prefill/Decode**: Are prefill and decode in the same worker or separate?
+   - *Aggregated*: Single worker handles both prefill and decode
+   - *Disaggregated*: Separate workers for prefill and decode, with KV cache transfer between them
+
+These combine into four deployment patterns:
+
+### EPD - Simple Aggregated
+
+All processing happens within a single worker - the simplest setup.
+
+```text
+HTTP Frontend (Rust)
+    ↓
+Worker (Python)
+    ↓ image load + encode + prefill + decode
+Response
+```
+
+| Component | Purpose |
+|-----------|---------|
+| Frontend (Rust) | HTTP entry point, tokenization, image URL preprocessing |
+| Worker | Complete inference pipeline (encode + prefill + decode) |
+
+**When to use:** Quick setup, smaller models, development/testing.
+
+### E/PD - Encode Separate
+
+Encoding happens in a separate worker; prefill and decode share the same engine.
+
+```text
+HTTP Frontend (Rust)
+    ↓
+Processor (Python)
+    ↓ tokenizes, extracts media URL
+Encode Worker (Python)
+    ↓ downloads media, generates embeddings, NIXL transfer
+PD Worker (Python)
+    ↓ receives embeddings via NIXL, prefill + decode
+Response
+```
+
+| Component | Purpose |
+|-----------|---------|
+| Frontend (Rust) | HTTP entry point |
+| Processor (Python) | Tokenization, extracts media URLs |
+| Encode Worker | Media encoding, embeddings generation |
+| PD Worker | Prefill + Decode with embeddings |
+
+**When to use:** Offload vision encoding to separate GPU, scale encode workers independently.
+
+### E/P/D - Full Disaggregation
+
+Full disaggregation with separate workers for encoding, prefill, and decode.
+There are two variants of this workflow:
+- Prefill-first, used by vLLM
+- Decode-first, used by SGlang
+
+Prefill-first:
+
+```text
+HTTP Frontend (Rust)
+    ↓
+Processor (Python)
+    ↓ tokenizes, extracts media URL
+Encode Worker (Python)
+    ↓ downloads media, generates embeddings, NIXL transfer
+Prefill Worker (Python)
+    ↓ receives embeddings via NIXL, prefill only, KV cache transfer
+Decode Worker (Python)
+    ↓ decode only, token generation
+Response
+```
+
+OR
+
+Decode-first:
+
+```text
+HTTP Frontend (Rust)
+    ↓
+Processor (Python)
+    ↓ tokenizes, extracts media URL
+Encode Worker (Python)
+    ↓ downloads media, generates embeddings, NIXL transfer
+Decode Worker (Python)
+    ↓ Bootstraps prefill worker
+Prefill Worker (Python)
+    ↓ receives embeddings via NIXL, prefill only, KV cache transfer
+Decode Worker (Python)
+    ↓ decode only, token generation
+Response
+```
+
+| Component | Purpose |
+|-----------|---------|
+| Frontend (Rust) | HTTP entry point |
+| Processor (Python) | Tokenization, extracts media URLs |
+| Encode Worker | Media encoding, embeddings generation |
+| Prefill Worker | Prefill only, transfers KV cache |
+| Decode Worker | Decode only, token generation |
+
+**When to use:** Maximum optimization, multi-node deployment, independent scaling of each phase.
+
+### EP/D - Traditional Disaggregated
+
+Encoding is combined with prefill, with decode separate.
+
+```text
+HTTP Frontend (Rust)
+    ↓
+Processor (Python)
+    ↓ tokenizes, extracts media URL
+Encode+Prefill Worker (Python)
+    ↓ downloads media, encodes inline, prefill, KV cache transfer
+Decode Worker (Python)
+    ↓ decode only, token generation
+Response
+```
+
+| Component | Purpose |
+|-----------|---------|
+| Frontend (Rust) | HTTP entry point |
+| Processor (Python) | Tokenization, extracts media URLs (vLLM only) |
+| Encode+Prefill Worker | Combined encoding and prefill |
+| Decode Worker | Decode only, token generation |
+
+> **Note:** TRT-LLM's EP/D mode skips the Python Processor - the Rust frontend handles tokenization and routes directly to the Prefill worker.
+> For multimodal requests, the Python prefill worker still re-tokenizes/builds inputs; Rust token_ids are ignored.
+
+**When to use:** Models without pre-computed embedding support (Llama 4), or TRT-LLM disaggregated deployment.
+
+## Example Workflows
+
+You can find example workflows and reference implementations for deploying multimodal models in:
+
+- [vLLM multimodal examples](https://github.com/ai-dynamo/dynamo/tree/main/examples/backends/vllm/launch)
+- [TRT-LLM multimodal examples](https://github.com/ai-dynamo/dynamo/tree/main/examples/backends/trtllm/launch)
+- [SGLang multimodal examples](https://github.com/ai-dynamo/dynamo/tree/main/examples/backends/sglang/launch)
+- [Advanced multimodal examples](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal/launch) (video, audio)

fern/pages/multimodal/sglang.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "SGLang Multimodal"
+---
+
+This document provides a comprehensive guide for multimodal inference using SGLang backend in Dynamo. SGLang multimodal uses specialized **E/PD or E/P/D** flows with **NIXL (RDMA)** for zero-copy tensor transfer.
+
+## Support Matrix
+
+| Modality | Input Format | Aggregated | Disaggregated | Notes |
+|----------|--------------|------------|---------------|-------|
+| **Image** | HTTP/HTTPS URL | Yes | Yes | Vision encoder generates embeddings |
+| **Image** | Data URL (Base64) | No | No |  |
+| **Video** | HTTP/HTTPS URL | No | No |  |
+| **Audio** | HTTP/HTTPS URL | No | No |  |
+
+### Supported URL Formats
+
+| Format | Example | Description |
+|--------|---------|-------------|
+| **HTTP/HTTPS** | `http://example.com/image.jpg` | Remote media files |
+
+## Deployment Patterns
+
+SGLang supports E/PD and E/P/D patterns only (always has a separate encode worker). See [Multimodal Architecture Patterns](index.md#architecture-patterns) for detailed explanations.
+
+| Pattern | Supported | Launch Script | Notes |
+|---------|-----------|---------------|-------|
+| EPD (Simple Aggregated) | ❌ | N/A | Not supported |
+| E/PD (Encode Separate) | ✅ | `multimodal_agg.sh` | Vision encoder separate |
+| E/P/D (Full Disaggregation) | ✅ | `multimodal_disagg.sh` | KV cache via bootstrap |
+| EP/D (Traditional Disaggregated) | ❌ | N/A | Not supported |
+
+### Component Flags
+
+| Component | Flag | Purpose |
+|-----------|------|---------|
+| Processor | `--multimodal-processor` | HTTP entry, OpenAI→SGLang conversion |
+| Encode Worker | `--multimodal-encode-worker` | Vision encoder, embeddings generation |
+| PD Worker | `--multimodal-worker` | Prefill + Decode with embeddings |
+| Decode Worker | `--multimodal-worker --serving-mode=decode` | Entry point for disaggregation |
+| Prefill Worker | `--multimodal-worker --serving-mode=prefill` | Called by Decode, bootstrap coordination |
+
+### SGLang-Specific Characteristics
+
+- **Vision Encoder in Python**: Encode worker loads vision model (AutoModel) and image processor (AutoImageProcessor)
+- **Token Expansion**: Single `<|image_pad|>` token replaced with N tokens based on embedding shape
+- **NIXL Transfer**: Embeddings transferred from Encoder → PD Worker using NIXL
+- **No Rust Processing**: All tokenization and image handling happens in Python
+
+## Use the Latest Release
+
+We recommend using the latest stable release of dynamo to avoid breaking changes:
+
+[![GitHub Release](https://img.shields.io/github/v/release/ai-dynamo/dynamo)](https://github.com/ai-dynamo/dynamo/releases/latest)
+
+You can find the [latest release](https://github.com/ai-dynamo/dynamo/releases/latest) and check out the corresponding branch with:
+
+```bash
+git checkout $(git describe --tags $(git rev-list --tags --max-count=1))
+```
+
+## E/PD Serving (Encode Separate)
+
+### Components
+
+- workers:
+  - [MultimodalEncodeWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/encode_worker_handler.py) for encoding
+  - [MultimodalWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for prefilling and decoding.
+- processor: [MultimodalProcessorHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/processor_handler.py)
+  - tokenizes the prompt using the chat template
+  - passes the text and image url to the MultimodalEncodeWorker.
+
+### Workflow
+
+The `MultimodalEncodeWorker` downloads and encodes the image and passes the embeddings to the MultimodalWorker. The work complete event is sent via NATS, while the embeddings tensor is transferred via RDMA through the NIXL interface. The `MultimodalWorker` then prefills and decodes the prompt in the same engine, as in the [LLM aggregated serving](../backends/sglang/README.md) example. Only the processor is registered to the Dynamo frontend as an available endpoint. Workers do NOT register - they are internal components and communicate via NATS.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --tokenized request + image_url--> encode_worker
+  encode_worker --request + embeddings--> worker
+
+  worker -.-> encode_worker
+  encode_worker -.-> processor
+  processor -.-> HTTP
+```
+
+
+### Launch
+
+```bash
+cd $DYNAMO_HOME/examples/backends/sglang
+./launch/multimodal_agg.sh
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe the image."
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
+            }
+          }
+        ]
+      }
+    ],
+    "max_tokens": 50,
+    "stream": false
+  }' | jq
+```
+
+## E/P/D Serving (Full Disaggregation)
+
+### Components
+
+- workers:
+  - [MultimodalEncodeWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/encode_worker_handler.py) for encoding
+  - [MultimodalWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for decoding
+  - [MultimodalPrefillWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py) for prefilling
+- processor: [MultimodalProcessorHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/sglang/request_handlers/multimodal/processor_handler.py) tokenizes the prompt and passes it to the MultimodalEncodeWorker.
+
+### Workflow
+
+In models like Qwen2.5-VL, embeddings are only required during the prefill stage. The image embeddings are transferred via NIXL from the Encode Worker to the Decode Worker (the entry point for disaggregation), which then coordinates with the Prefill Worker. The Prefill Worker processes the embeddings and forwards the KV cache back to the Decode Worker for token generation.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --tokenized request + image_url--> encode_worker
+  encode_worker --request + embeddings--> worker
+  worker --request + embeddings--> prefill_worker
+
+  prefill_worker --KV Cache--> worker
+  encode_worker -.-> processor
+  worker -.-> encode_worker
+  processor -.-> HTTP
+```
+
+### Launch
+
+```bash
+cd $DYNAMO_HOME/examples/backends/sglang
+./launch/multimodal_disagg.sh
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe the image."
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
+            }
+          }
+        ]
+      }
+    ],
+    "max_tokens": 50,
+    "stream": false
+  }' | jq
+```
+
+## Bootstrap Coordination
+
+SGLang disaggregation uses a bootstrap mechanism for P->D coordination:
+
+### Request Flow (Important)
+
+```text
+Client → Frontend → Processor → Encode → DECODE Worker → Prefill Worker
+                                               ↑
+                                    Entry point for disaggregation!
+```
+
+### Bootstrap Process
+
+1. **Decode Worker** receives request from Encode Worker
+2. **Decode Worker** calls Prefill Worker via NATS to request bootstrap info
+3. **Prefill Worker** generates `{host, port, room}` and returns immediately
+4. **Both workers** connect to same "room" using bootstrap coordinates
+5. **SGLang internally** transfers KV cache state via bootstrap connection (not NIXL)
+
+### Key Difference from vLLM
+
+- vLLM: Frontend → Prefill → Decode (Prefill is entry point)
+- SGLang: Frontend → Processor → Encode → **Decode → Prefill** (Decode is entry point)
+
+## Inter-Component Communication
+
+### Control Flow (NATS)
+
+All component-to-component communication happens via NATS:
+
+#### E/PD Mode (Encode Separate)
+
+```text
+Processor → Encode Worker → PD Worker
+  (NATS)        (NATS + NIXL embeddings)
+```
+
+#### E/P/D Mode (Full Disaggregation)
+
+```text
+Processor → Encode Worker → DECODE Worker → Prefill Worker
+  (NATS)        (NATS)            (NATS)
+                             ↓
+                    Decode requests bootstrap
+                             ↓
+                    Prefill returns {host, port, room}
+                             ↓
+                    Both connect via bootstrap
+                             ↓
+                    SGLang internal KV cache transfer
+```
+
+### Detailed Message Flow
+
+```text
+Processor → Encode Worker:
+  - NATS round_robin with SglangMultimodalRequest
+  - Contains: tokenized input_ids, image URL, sampling params
+
+Encode Worker → Decode/PD Worker:
+  - NATS round_robin to "backend" component
+  - Contains: expanded token_ids, NIXL metadata, embeddings shape
+  - NIXL transfer: embeddings tensor
+
+Decode Worker → Prefill Worker (disagg only):
+  - NATS call to "prefill" component
+  - Decode requests bootstrap coordinates
+  - Prefill returns: {bootstrap_host, bootstrap_port, bootstrap_room}
+
+Prefill ↔ Decode (via bootstrap):
+  - SGLang internal connection (not NATS)
+  - KV cache state shared via bootstrap mechanism
+```
+
+### Data Transfer (NIXL)
+
+NIXL is used only for embedding transfer:
+
+```python
+# Encode Worker
+descriptor = connect.Descriptor(precomputed_embeddings)
+with connector.create_readable(descriptor) as readable:
+    request.serialized_request = readable.metadata()
+    await pd_worker_client.round_robin(request)
+    await readable.wait_for_completion()
+
+# PD Worker
+embeddings = torch.empty(request.embeddings_shape, dtype=torch.float16)
+descriptor = connect.Descriptor(embeddings)
+read_op = await connector.begin_read(request.serialized_request, descriptor)
+await read_op.wait_for_completion()
+```
+
+## Vision Encoding Details
+
+### Encode Worker Components
+
+The encode worker loads and runs the vision model in Python:
+
+```python
+self.image_processor = AutoImageProcessor.from_pretrained(
+    model_path, trust_remote_code=True
+)
+self.vision_model = AutoModel.from_pretrained(
+    model_path,
+    device_map="auto",
+    torch_dtype=torch.float16,
+    trust_remote_code=True
+)
+```
+
+### Token Expansion Process
+
+1. Processor inserts single image token (e.g., `<|image_pad|>`)
+2. Encode worker generates embeddings: `shape = (batch, num_patches, hidden_dim)`
+3. Encode worker replaces single token with `num_patches` tokens
+4. Downstream worker receives expanded token sequence
+
+Example:
+
+```python
+# Before: ["Hello", "<|image_pad|>", "world"]
+# After:  ["Hello", "<|image_pad|>", "<|image_pad|>", ...(576 tokens), "world"]
+```
+
+## Chat Template Processing
+
+SGLang uses its own chat template system:
+
+```python
+from sglang.srt.parser.conversation import chat_templates
+
+conv = chat_templates["qwen2-vl"].copy()
+conv.append_message(conv.roles[0], f"{conv.image_token} Describe this image")
+processed = tokenizer(text=conv.get_prompt(), return_tensors="pt")
+```
+
+Supported templates: `qwen2-vl`, `llama-3`, `vicuna`, etc.
+
+## NIXL Usage
+
+| Use Case | NIXL Used? | Data Transfer | Notes |
+|----------|------------|---------------|-------|
+| E/PD (Encode Separate) | Yes | Encoder → PD (embeddings) | Vision encoder separate |
+| E/P/D (Full Disaggregation) | Yes | Encoder → Prefill (embeddings) | KV cache via SGLang bootstrap |
+
+**Key Difference:** SGLang P/D uses bootstrap mechanism, not NIXL for KV cache like vLLM.
+
+## Known Limitations
+
+- **No Data URL support** - Only HTTP/HTTPS URLs supported; `data:image/...` base64 URLs not supported
+- **No pre-computed embeddings** - Cannot use `.pt`, `.pth`, `.bin` embedding files; vision encoder runs for every request
+- **No video support** - No video encoder implementation
+- **No audio support** - No audio encoder implementation
+- **Only Processor registers with Dynamo** - Workers are internal components, frontend routes to Processor only
+- **Disaggregated routing** - Decode Worker is the entry point (calls Prefill), cannot route directly to Prefill workers
+- **Limited model generalization** - Token expansion logic is model-specific; adding new models may require implementation updates
+
+## Supported Models
+
+SGLang multimodal **only supports image-based vision-language models**:
+
+- **Qwen2-VL** / **Qwen2.5-VL** (primary support)
+- Models with `AutoImageProcessor` and vision tower
+- Models compatible with SGLang's image embedding format
+
+## Key Files
+
+| File | Description |
+|------|-------------|
+| `components/src/dynamo/sglang/main.py` | Component initialization, only Processor registers |
+| `components/src/dynamo/sglang/request_handlers/multimodal/processor_handler.py` | Processor implementation, OpenAI→SGLang |
+| `components/src/dynamo/sglang/request_handlers/multimodal/encode_worker_handler.py` | Vision encoder, embeddings generation |
+| `components/src/dynamo/sglang/request_handlers/multimodal/worker_handler.py` | PD/Prefill/Decode workers, NIXL read |
+| `components/src/dynamo/sglang/multimodal_utils/multimodal_chat_processor.py` | Chat template processing |
+| `components/src/dynamo/sglang/protocol.py` | Request/response data structures |
+| `components/src/dynamo/sglang/register.py` | Registration logic (only called for Processor) |

fern/pages/multimodal/trtllm.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "TensorRT-LLM Multimodal"
+---
+
+This document provides a comprehensive guide for multimodal inference using TensorRT-LLM backend in Dynamo.
+
+You can provide multimodal inputs in the following ways:
+- By sending image URLs
+- By providing paths to pre-computed embedding files
+
+> **Note:** You should provide **either image URLs or embedding file paths** in a single request.
+
+## Support Matrix
+
+| Modality | Input Format | Aggregated | Disaggregated | Notes |
+|----------|--------------|------------|---------------|-------|
+| **Image** | HTTP/HTTPS URL | Yes | Yes | Full support for all image models |
+| **Image** | Pre-computed Embeddings (.pt, .pth, .bin) | Yes | Yes | Direct embedding files |
+| **Video** | HTTP/HTTPS URL | No | No | Not implemented |
+| **Audio** | HTTP/HTTPS URL | No | No | Not implemented |
+
+### Supported URL Formats
+
+| Format | Example | Description |
+|--------|---------|-------------|
+| **HTTP/HTTPS** | `http://example.com/image.jpg` | Remote media files |
+| **Pre-computed Embeddings** | `/path/to/embedding.pt` | Local embedding files (.pt, .pth, .bin) |
+
+## Deployment Patterns
+
+TRT-LLM supports aggregated and traditional disaggregated patterns. See [Architecture Patterns](index.md#architecture-patterns) for detailed explanations.
+
+| Pattern | Supported | Launch Script | Notes |
+|---------|-----------|---------------|-------|
+| EPD (Simple Aggregated) | ✅ | `agg.sh` | Easiest setup |
+| E/PD (Encode Separate) | ❌ | N/A | Not supported |
+| E/P/D (Full Disaggregation) | 🚧 WIP | N/A | PR #4668 in progress |
+| EP/D (Traditional Disaggregated) | ✅ | `disagg_multimodal.sh` | Prefill handles encoding |
+
+### Component Flags
+
+| Component | Flag | Purpose |
+|-----------|------|---------|
+| Worker | `--modality multimodal` | Complete pipeline (aggregated) |
+| Prefill Worker | `--disaggregation-mode prefill` | Image processing + Prefill (multimodal tokenization happens here) |
+| Decode Worker | `--disaggregation-mode decode` | Decode only |
+| Encode Worker (WIP) | `--disaggregation-mode encode` | Image encoding (E/P/D flow) |
+
+## Aggregated Serving
+
+Quick steps to launch Llama-4 Maverick BF16 in aggregated mode:
+
+```bash
+cd $DYNAMO_HOME
+
+export AGG_ENGINE_ARGS=./examples/backends/trtllm/engine_configs/llama4/multimodal/agg.yaml
+export SERVED_MODEL_NAME="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
+export MODEL_PATH="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
+./examples/backends/trtllm/launch/agg.sh
+```
+
+**Client:**
+```bash
+curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
+    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Describe the image"
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
+                    }
+                }
+            ]
+        }
+    ],
+    "stream": false,
+    "max_tokens": 160
+}'
+```
+
+## Disaggregated Serving
+
+Example using `Qwen/Qwen2-VL-7B-Instruct`:
+
+```bash
+cd $DYNAMO_HOME
+
+export MODEL_PATH="Qwen/Qwen2-VL-7B-Instruct"
+export SERVED_MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
+export PREFILL_ENGINE_ARGS="examples/backends/trtllm/engine_configs/qwen2-vl-7b-instruct/prefill.yaml"
+export DECODE_ENGINE_ARGS="examples/backends/trtllm/engine_configs/qwen2-vl-7b-instruct/decode.yaml"
+export MODALITY="multimodal"
+
+./examples/backends/trtllm/launch/disagg.sh
+```
+
+```bash
+curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
+    "model": "Qwen/Qwen2-VL-7B-Instruct",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "Describe the image"
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
+                    }
+                }
+            ]
+        }
+    ],
+    "stream": false,
+    "max_tokens": 160
+}'
+```
+
+For a large model like `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, a multi-node setup is required for disaggregated serving (see [Multi-node Deployment](#multi-node-deployment-slurm) below), while aggregated serving can run on a single node. This is because the model with a disaggregated configuration is too large to fit on a single node's GPUs. For instance, running this model in disaggregated mode requires 2 nodes with 8xH200 GPUs or 4 nodes with 4xGB200 GPUs.
+
+## Pre-computed Embeddings with E/P/D Flow
+
+For high-performance multimodal inference, Dynamo supports pre-computed embeddings with an **Encode-Prefill-Decode (E/P/D)** flow using **NIXL (RDMA)** for zero-copy tensor transfer.
+
+### Supported File Types
+
+- `.pt` - PyTorch tensor files
+- `.pth` - PyTorch checkpoint files
+- `.bin` - Binary tensor files
+
+### Embedding File Formats
+
+TRT-LLM supports two formats for embedding files:
+
+**1. Simple Tensor Format**
+
+Direct tensor saved as `.pt` file containing only the embedding tensor:
+
+```python
+embedding_tensor = torch.rand(1, 576, 4096)  # [batch, seq_len, hidden_dim]
+torch.save(embedding_tensor, "embedding.pt")
+```
+
+**2. Dictionary Format with Auxiliary Data**
+
+Dictionary containing multiple keys, used by models like Llama-4 that require additional metadata:
+
+```python
+embedding_dict = {
+    "mm_embeddings": torch.rand(1, 576, 4096),
+    "special_tokens": [128256, 128257],
+    "image_token_offsets": [[0, 576]],
+    # ... other model-specific metadata
+}
+torch.save(embedding_dict, "llama4_embedding.pt")
+```
+
+- **Simple tensors**: Loaded directly and passed to `mm_embeddings` parameter
+- **Dictionary format**: `mm_embeddings` key extracted as main tensor, other keys preserved as auxiliary data
+
+### How to Launch
+
+```bash
+cd $DYNAMO_HOME/examples/backends/trtllm
+
+# Launch 3-worker E/P/D flow with NIXL
+./launch/epd_disagg.sh
+```
+
+> **Note:** This script is designed for 8-node H200 with `Llama-4-Scout-17B-16E-Instruct` model and assumes you have a model-specific embedding file ready.
+
+### Configuration
+
+```bash
+# Encode endpoint for Prefill → Encode communication
+export ENCODE_ENDPOINT="dyn://dynamo.tensorrt_llm_encode.generate"
+
+# Security: Allowed directory for embedding files (default: /tmp)
+export ALLOWED_LOCAL_MEDIA_PATH="/tmp"
+
+# Security: Max file size to prevent DoS attacks (default: 50MB)
+export MAX_FILE_SIZE_MB=50
+```
+
+### Example Request with Pre-computed Embeddings
+
+```bash
+curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
+    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Describe the image"},
+                {"type": "image_url", "image_url": {"url": "/path/to/embedding.pt"}}
+            ]
+        }
+    ],
+    "max_tokens": 160
+}'
+```
+
+### E/P/D Architecture
+
+The E/P/D flow implements a **3-worker architecture**:
+
+- **Encode Worker**: Loads pre-computed embeddings, transfers via NIXL
+- **Prefill Worker**: Receives embeddings, handles context processing and KV-cache generation
+- **Decode Worker**: Performs streaming token generation
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Frontend
+    participant PrefillWorker as "Prefill Worker"
+    participant EncodeWorker as "Encode Worker"
+    participant DecodeWorker as "Decode Worker"
+    participant NIXL as "NIXL (RDMA)"
+
+    Client->>Frontend: POST /v1/chat/completions
+    Frontend->>PrefillWorker: Route to prefill worker
+    PrefillWorker->>EncodeWorker: Send request (embedding paths)
+    EncodeWorker->>NIXL: Create readable operation
+    EncodeWorker->>PrefillWorker: Send metadata + NIXL info
+    PrefillWorker->>NIXL: Begin read operation
+    NIXL-->>PrefillWorker: Zero-copy transfer complete
+    PrefillWorker->>Frontend: Return prefill response
+    Frontend->>DecodeWorker: Route to decode worker
+    DecodeWorker->>Frontend: Stream response chunks
+    Frontend->>Client: Stream response
+```
+
+## Multi-node Deployment (Slurm)
+
+This section demonstrates how to deploy large multimodal models that require a multi-node setup using Slurm.
+
+> **Note:** The scripts referenced in this section can be found in [`examples/basics/multinode/trtllm/`](https://github.com/ai-dynamo/dynamo/tree/main/examples/basics/multinode/trtllm/).
+
+### Environment Setup
+
+Assuming you have allocated your nodes via `salloc` and are inside an interactive shell:
+
+```bash
+# Container image (build using docs/backends/trtllm/README.md#build-container)
+export IMAGE="<dynamo_trtllm_image>"
+
+# Host:container path pairs for mounting
+export MOUNTS="${PWD}/../../../../:/mnt"
+
+# Model configuration
+export MODEL_PATH="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
+export SERVED_MODEL_NAME="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
+export MODALITY=${MODALITY:-"multimodal"}
+```
+
+### Multi-node Disaggregated Launch
+
+For 4 4xGB200 nodes (2 for prefill, 2 for decode):
+
+```bash
+# Customize parallelism to match your engine configs
+# export PREFILL_ENGINE_CONFIG="/mnt/examples/backends/trtllm/engine_configs/llama4/multimodal/prefill.yaml"
+# export DECODE_ENGINE_CONFIG="/mnt/examples/backends/trtllm/engine_configs/llama4/multimodal/decode.yaml"
+# export NUM_PREFILL_NODES=2
+# export NUM_DECODE_NODES=2
+# export NUM_GPUS_PER_NODE=4
+
+# Launches frontend + etcd/nats on head node, plus prefill and decode workers
+./srun_disaggregated.sh
+```
+
+### Understanding the Output
+
+1. `srun_disaggregated.sh` launches three srun jobs: frontend, prefill worker, and decode worker
+2. The OpenAI frontend will dynamically discover workers as they register:
+   ```
+   INFO dynamo_run::input::http: Watching for remote model at models
+   INFO dynamo_llm::http::service::service_v2: Starting HTTP service on: 0.0.0.0:8000
+   ```
+3. TRT-LLM workers output progress from each MPI rank while loading
+4. When ready, the frontend logs:
+   ```
+   INFO dynamo_llm::discovery::watcher: added model model_name="meta-llama/Llama-4-Maverick-17B-128E-Instruct"
+   ```
+
+### Cleanup
+
+```bash
+pkill srun
+```
+
+## NIXL Usage
+
+| Use Case | Script | NIXL Used? | Data Transfer |
+|----------|--------|------------|---------------|
+| EPD (Simple Aggregated) | `agg.sh` | No | All in one worker |
+| EP/D (Traditional Disaggregated) | `disagg_multimodal.sh` | Optional | Prefill → Decode (KV cache via UCX or NIXL) |
+| E/P/D (pre-computed embeddings) | `epd_disagg.sh` | Yes | Encoder → Prefill (embeddings via NIXL) |
+| E/P/D (WIP) | N/A | No | Encoder → Prefill (handles via params), Prefill → Decode (KV cache) |
+
+> **Note:** NIXL for KV cache transfer is currently beta and only supported on AMD64 (x86_64) architecture.
+
+## ModelInput Types and Registration
+
+TRT-LLM workers register with Dynamo using:
+
+| ModelInput Type | Preprocessing | Use Case |
+|-----------------|---------------|----------|
+| `ModelInput.Tokens` | Rust frontend may tokenize, but multimodal flows re-tokenize and build inputs in the Python worker; Rust token_ids are ignored | All TRT-LLM workers |
+
+```python
+# TRT-LLM Worker - Register with Tokens
+await register_llm(
+    ModelInput.Tokens,      # Rust does minimal preprocessing
+    model_type,             # ModelType.Chat or ModelType.Prefill
+    generate_endpoint,
+    model_name,
+    ...
+)
+```
+
+## Inter-Component Communication
+
+| Transfer Stage | Message | NIXL Transfer |
+|----------------|---------|---------------|
+| **Frontend → Prefill** | Request with image URL or embedding path | No |
+| **Encode → Prefill (pre-computed)** | NIXL metadata | Yes (Embeddings tensor) |
+| **Encode → Prefill (Image URL) (WIP)** | Disaggregated params with multimodal handles | No |
+| **Prefill → Decode** | Disaggregated params | Configurable (KV cache: NIXL default, UCX optional) |
+
+## Known Limitations
+
+- **No Data URL support** - Only HTTP/HTTPS URLs supported; `data:image/...` base64 URLs not supported
+- **No video support** - No video encoder implementation
+- **No audio support** - No audio encoder implementation
+- **Multimodal preprocessing/tokenization happens in Python** - Rust may forward token_ids, but multimodal requests are parsed and re-tokenized in the Python worker
+- **E/P/D mode is WIP** - Full E/P/D with image URLs under development
+- **Multi-node H100 limitation** - Loading `meta-llama/Llama-4-Maverick-17B-128E-Instruct` with 8 nodes of H100 with TP=16 is not possible due to head count divisibility (`num_attention_heads: 40` not divisible by `tp_size: 16`)
+
+## Supported Models
+
+Multimodal models listed in [TensorRT-LLM supported models](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/supported-models.md) are supported by Dynamo.
+
+Common examples:
+- Llama 4 Vision models (Maverick, Scout)
+- Qwen2-VL models
+- Other vision-language models with TRT-LLM support
+
+## Key Files
+
+| File | Description |
+|------|-------------|
+| `components/src/dynamo/trtllm/main.py` | Worker initialization and setup |
+| `components/src/dynamo/trtllm/utils/trtllm_utils.py` | Command-line argument parsing |
+| `components/src/dynamo/trtllm/multimodal_processor.py` | Multimodal request processing |
+| `components/src/dynamo/trtllm/request_handlers/handlers.py` | Request handler factory |
+| `components/src/dynamo/trtllm/request_handlers/handler_base.py` | Base handler and disaggregation modes |
+

fern/pages/multimodal/vllm.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "vLLM Multimodal"
+---
+
+This document provides a comprehensive guide for multimodal inference using vLLM backend in Dynamo.
+
+<Warning>
+**Security Requirement**: All multimodal workers require the `--enable-multimodal` flag to be explicitly set at startup. This is a security feature to prevent unintended processing of multimodal data from untrusted sources. Workers will fail at startup if multimodal flags (e.g., `--multimodal-worker`, `--multimodal-processor`) are used without `--enable-multimodal`.
+This flag is analogous to `--enable-mm-embeds` in vllm serve but also extends it to all multimodal content (url, embeddings, b64).
+</Warning>
+
+## Support Matrix
+
+| Modality | Input Format | Aggregated | Disaggregated | Notes |
+|----------|--------------|------------|---------------|-------|
+| **Image** | HTTP/HTTPS URL | Yes | Yes | Full support for all image models |
+| **Image** | Data URL (Base64) | Yes | Yes | Inline base64-encoded images |
+| **Video** | HTTP/HTTPS URL | Yes | Yes | Frame extraction and processing |
+| **Audio** | HTTP/HTTPS URL | Yes | Yes | Experimental - requires audio dependencies |
+
+### Supported URL Formats
+
+| Format | Example | Description |
+|--------|---------|-------------|
+| **HTTP/HTTPS** | `http://example.com/image.jpg` | Remote media files |
+| **Data URL** | `data:image/jpeg;base64,/9j/4AAQ...` | Base64-encoded inline data |
+
+## Deployment Patterns
+
+vLLM supports all multimodal deployment patterns. See [Architecture Patterns](index.md#architecture-patterns) for detailed explanations.
+
+| Pattern | Supported | Launch Script | Notes |
+|---------|-----------|---------------|-------|
+| EPD (Simple Aggregated) | ✅ | `agg_multimodal.sh` | Easiest setup |
+| E/PD (Encode Separate) | ✅ | `agg_multimodal_epd.sh` | Separate encode worker |
+| E/P/D (Full Disaggregation) | ✅ | `disagg_multimodal_epd.sh` | All stages separate |
+| EP/D (Traditional Disaggregated) | ✅ | `disagg_multimodal_llama.sh` | For Llama 4 models |
+| E/PD (EC Connector) | ✅ | `agg_multimodal_ec_connector.sh` | vLLM-native encoder with ECConnector |
+
+### Component Flags
+
+| Component | Flag | Purpose |
+|-----------|------|---------|
+| Processor | `--multimodal-processor` | HTTP entry, tokenization |
+| Encode Worker | `--multimodal-encode-worker` | Media encoding |
+| PD Worker | `--multimodal-worker` | Prefill + Decode |
+| Prefill Worker | `--multimodal-worker --is-prefill-worker` | Prefill only |
+| Decode Worker | `--multimodal-decode-worker` | Decode only |
+| Encode+Prefill Worker | `--multimodal-encode-prefill-worker --is-prefill-worker` | Combined (Llama 4) |
+| vLLM Native Encoder | `--vllm-native-encoder-worker` | vLLM-native encoding with ECConnector |
+
+## Use the Latest Release
+
+We recommend using the latest stable release of dynamo to avoid breaking changes:
+
+[![GitHub Release](https://img.shields.io/github/v/release/ai-dynamo/dynamo)](https://github.com/ai-dynamo/dynamo/releases/latest)
+
+You can find the [latest release](https://github.com/ai-dynamo/dynamo/releases/latest) and check out the corresponding branch with:
+
+```bash
+git checkout $(git describe --tags $(git rev-list --tags --max-count=1))
+```
+
+## Image Serving
+
+### E/PD Serving (Encode Separate)
+
+**Components:**
+
+- workers: [EncodeWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py) for encoding and [MultimodalPDWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/vllm/multimodal_handlers/worker_handler.py) for prefilling and decoding.
+- processor: Tokenizes the prompt and passes it to the EncodeWorkerHandler.
+- frontend: HTTP endpoint to handle incoming requests.
+
+**Workflow:**
+
+The EncodeWorkerHandler encodes the image and passes the embeddings to the MultimodalPDWorkerHandler via NATS and RDMA. The work complete event is sent via NATS, while the embeddings tensor is transferred via RDMA through the NIXL interface.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --image_url--> encode_worker
+  encode_worker --> processor
+  encode_worker --embeddings--> pd_worker
+  pd_worker --> encode_worker
+```
+
+> **Note:** Aggregated serving supports LLaVA 1.5 7B and Qwen2.5-VL-7B-Instruct. Disaggregated serving is currently only confirmed for LLaVA.
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+# Serve a LLaVA 1.5 7B model:
+bash launch/agg_multimodal_epd.sh --model llava-hf/llava-1.5-7b-hf
+# Serve a Qwen2.5-VL model:
+bash launch/agg_multimodal_epd.sh --model Qwen/Qwen2.5-VL-7B-Instruct
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+      "model": "llava-hf/llava-1.5-7b-hf",
+      "messages": [
+        {
+          "role": "user",
+          "content": [
+            {
+              "type": "text",
+              "text": "What is in this image?"
+            },
+            {
+              "type": "image_url",
+              "image_url": {
+                "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
+              }
+            }
+          ]
+        }
+      ],
+      "max_tokens": 300,
+      "temperature": 0.0,
+      "stream": false
+    }'
+```
+
+### E/P/D Serving (Full Disaggregation)
+
+**Components:**
+
+- workers: [EncodeWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py) for encoding, [MultimodalDecodeWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/vllm/multimodal_handlers/worker_handler.py) for decoding, and [MultimodalPDWorkerHandler](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/vllm/multimodal_handlers/worker_handler.py) for prefilling.
+- processor: Tokenizes the prompt and passes it to the EncodeWorkerHandler.
+- frontend: HTTP endpoint to handle incoming requests.
+
+**Workflow:**
+
+For the LLaVA model, embeddings are only required during the prefill stage. The EncodeWorkerHandler is connected directly to the prefill worker, encoding the image and passing embeddings via NATS and RDMA. The prefill worker performs the prefilling step and forwards the KV cache to the decode worker.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --image_url--> encode_worker
+  encode_worker --> processor
+  encode_worker --embeddings--> prefill_worker
+  prefill_worker --> encode_worker
+  prefill_worker --> decode_worker
+  decode_worker --> prefill_worker
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+bash launch/disagg_multimodal_epd.sh --model llava-hf/llava-1.5-7b-hf
+```
+
+<Note>
+Disaggregation is currently only confirmed to work with LLaVA. Qwen2.5-VL is not confirmed to be supported.
+</Note>
+
+## ECConnector Serving
+
+ECConnector is vLLM's native connector for transferring multimodal embeddings via an Embedding Cache. The encoder worker acts as a **producer** (writes embeddings), while the PD worker acts as a **consumer** (reads embeddings).
+
+**Workflow:**
+
+```mermaid
+flowchart LR
+  HTTP --> processor[EC Processor]
+  processor --image_url--> encoder[vLLM Native Encoder<br/>Producer]
+  encoder --writes--> cache[(Embedding Cache)]
+  cache --reads--> pd[PD Worker<br/>Consumer]
+  pd --> processor
+  processor --> HTTP
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+bash launch/agg_multimodal_ec_connector.sh --model llava-hf/llava-1.5-7b-hf
+
+# Custom storage path for Embedding Cache
+bash launch/agg_multimodal_ec_connector.sh --ec-storage-path /shared/encoder-cache
+```
+
+**Client:** Same as [E/PD Serving](#epd-serving-encode-separate)
+
+## Llama 4 Serving
+
+The Llama 4 model family is natively multimodal. Unlike LLaVA, they do not directly consume image embeddings as input (see the [vLLM support matrix](https://docs.vllm.ai/en/latest/models/supported_models.html#text-generation_1)). Therefore, the encoder worker is not used and encoding is done alongside prefill.
+
+Example model: `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8` on H100x8.
+
+### Llama 4 Aggregated Serving
+
+**Workflow:**
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --image_url--> pd_worker
+  pd_worker --> processor
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+bash launch/agg_multimodal_llama.sh
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+      "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
+      "messages": [
+        {
+          "role": "user",
+          "content": [
+            {
+              "type": "text",
+              "text": "What is in this image?"
+            },
+            {
+              "type": "image_url",
+              "image_url": {
+                "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
+              }
+            }
+          ]
+        }
+      ],
+      "max_tokens": 300,
+      "temperature": 0.0,
+      "stream": false
+    }'
+```
+
+### Llama 4 Disaggregated Serving
+
+**Workflow:**
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --image_url--> prefill_worker
+  prefill_worker --> processor
+  prefill_worker --> decode_worker
+  decode_worker --> prefill_worker
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/backends/vllm
+bash launch/disagg_multimodal_llama.sh --head-node
+
+# On a separate node with NATS_SERVER and ETCD_ENDPOINTS pointing to head node:
+cd $DYNAMO_HOME/examples/backends/vllm
+bash launch/disagg_multimodal_llama.sh
+```
+
+## Video Serving
+
+### Video Aggregated Serving
+
+**Components:**
+
+- workers: [VideoEncodeWorker](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal/components/video_encode_worker.py) for decoding video into frames, and [VllmPDWorker](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal/components/worker.py) for prefilling and decoding.
+- processor: Tokenizes the prompt and passes it to the VideoEncodeWorker.
+- frontend: HTTP endpoint to handle incoming requests.
+
+**Workflow:**
+
+The VideoEncodeWorker decodes the video into frames. Unlike the image pipeline which generates embeddings, this pipeline passes raw frames directly to the VllmPDWorker via NATS and RDMA.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --video_url--> video_encode_worker
+  video_encode_worker --> processor
+  video_encode_worker --frames--> pd_worker
+  pd_worker --> video_encode_worker
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/multimodal
+bash launch/video_agg.sh
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+      "model": "llava-hf/LLaVA-NeXT-Video-7B-hf",
+      "messages": [
+        {
+          "role": "user",
+          "content": [
+            {
+              "type": "text",
+              "text": "Describe the video in detail"
+            },
+            {
+              "type": "video_url",
+              "video_url": {
+                "url": "https://storage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"
+              }
+            }
+          ]
+        }
+      ],
+      "max_tokens": 300,
+      "stream": false
+    }' | jq
+```
+
+### Video Disaggregated Serving
+
+**Workflow:**
+
+For the LLaVA-NeXT-Video-7B model, frames are only required during the prefill stage. The VideoEncodeWorker is connected directly to the prefill worker, decoding the video into frames and passing them via RDMA.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --video_url--> video_encode_worker
+  video_encode_worker --> processor
+  video_encode_worker --frames--> prefill_worker
+  prefill_worker --> video_encode_worker
+  prefill_worker --> decode_worker
+  decode_worker --> prefill_worker
+```
+
+**Launch:**
+
+```bash
+cd $DYNAMO_HOME/examples/multimodal
+bash launch/video_disagg.sh
+```
+
+## Audio Serving
+
+### Audio Aggregated Serving
+
+**Components:**
+
+- workers: [AudioEncodeWorker](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal/components/audio_encode_worker.py) for decoding audio into embeddings, and [VllmPDWorker](https://github.com/ai-dynamo/dynamo/tree/main/examples/multimodal/components/worker.py) for prefilling and decoding.
+- processor: Tokenizes the prompt and passes it to the AudioEncodeWorker.
+- frontend: HTTP endpoint to handle incoming requests.
+
+**Workflow:**
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --audio_url--> audio_encode_worker
+  audio_encode_worker --> processor
+  audio_encode_worker --embeddings--> pd_worker
+  pd_worker --> audio_encode_worker
+```
+
+**Launch:**
+
+```bash
+pip install vllm["audio"] accelerate # multimodal audio models dependency
+cd $DYNAMO_HOME/examples/multimodal
+bash launch/audio_agg.sh
+```
+
+**Client:**
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+      "model": "Qwen/Qwen2-Audio-7B-Instruct",
+      "messages": [
+        {
+          "role": "user",
+          "content": [
+            {
+              "type": "text",
+              "text": "What is recited in the audio?"
+            },
+            {
+              "type": "audio_url",
+              "audio_url": {
+                "url": "https://raw.githubusercontent.com/yuekaizhang/Triton-ASR-Client/main/datasets/mini_en/wav/1221-135766-0002.wav"
+              }
+            }
+          ]
+        }
+      ],
+      "max_tokens": 6000,
+      "temperature": 0.8,
+      "stream": false
+    }' | jq
+```
+
+### Audio Disaggregated Serving
+
+**Workflow:**
+
+For the Qwen2-Audio model, audio embeddings are only required during the prefill stage. The AudioEncodeWorker is connected directly to the prefill worker.
+
+```mermaid
+flowchart LR
+  HTTP --> processor
+  processor --> HTTP
+  processor --audio_url--> audio_encode_worker
+  audio_encode_worker --> processor
+  audio_encode_worker --embeddings--> prefill_worker
+  prefill_worker --> audio_encode_worker
+  prefill_worker --> decode_worker
+  decode_worker --> prefill_worker
+```
+
+**Launch:**
+
+```bash
+pip install vllm["audio"] accelerate # multimodal audio models dependency
+cd $DYNAMO_HOME/examples/multimodal
+bash launch/audio_disagg.sh
+```
+
+## NIXL Usage
+
+| Use Case | Script | NIXL Used? | Data Transfer |
+|----------|--------|------------|---------------|
+| EPD (Simple Aggregated) | `agg_multimodal.sh` | No | All in one worker |
+| E/PD (Encode Separate) | `agg_multimodal_epd.sh` | Yes | Encoder → PD (embeddings) |
+| E/P/D (Full Disaggregation) | `disagg_multimodal_epd.sh` | Yes | Encoder → Prefill (embeddings), Prefill → Decode (KV cache) |
+| EP/D (Llama 4) | `disagg_multimodal_llama.sh` | Yes | Prefill → Decode (KV cache) |
+| E/PD (EC Connector) | `agg_multimodal_ec_connector.sh` | No | ECConnector via Embedding Cache |
+
+## ModelInput Types and Registration
+
+Dynamo's Rust SDK supports two input types that determine how the HTTP frontend preprocesses requests:
+
+| ModelInput Type | Preprocessing | Use Case |
+|-----------------|---------------|----------|
+| `ModelInput.Text` | None (raw text passed through) | Components that tokenize themselves |
+| `ModelInput.Tokens` | Rust SDK would tokenize (but bypassed in multimodal) | Components expecting pre-tokenized input |
+
+**Registration Pattern:**
+
+```python
+# Processor - Entry point from HTTP frontend
+await register_llm(
+    ModelInput.Text,        # Frontend sends raw text
+    ModelType.Chat,
+    generate_endpoint,
+    model_name,
+    ...
+)
+
+# Workers - Internal components
+await register_llm(
+    ModelInput.Tokens,      # Expect pre-tokenized input
+    ModelType.Chat,         # or ModelType.Prefill for prefill workers
+    generate_endpoint,
+    model_name,
+    ...
+)
+```
+
+## Known Limitations
+
+- **Disaggregated flows require Python Processor** - All multimodal disaggregation requires the Python Processor component (`ModelInput.Text`).
+
+## Supported Models
+
+The following models have been tested with Dynamo's vLLM multimodal backend:
+
+- **Qwen2.5-VL** - `Qwen/Qwen2.5-VL-7B-Instruct`
+- **Qwen3-VL** - `Qwen/Qwen3-VL-30B-A3B-Instruct-FP8`
+- **LLaVA 1.5** - `llava-hf/llava-1.5-7b-hf`
+- **Llama 4 Maverick** - `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`
+- **LLaVA Next Video** - `llava-hf/LLaVA-NeXT-Video-7B-hf`
+- **Qwen2-Audio** - `Qwen/Qwen2-Audio-7B-Instruct`
+
+For a complete list of multimodal models supported by vLLM, see [vLLM Supported Multimodal Models](https://docs.vllm.ai/en/latest/models/supported_models/#list-of-multimodal-language-models). Models listed there should work with Simple Aggregated Mode but may not be explicitly tested.
+
+## Key Files
+
+| File | Description |
+|------|-------------|
+| `components/src/dynamo/vllm/main.py` | Worker initialization and setup |
+| `components/src/dynamo/vllm/args.py` | Command-line argument parsing |
+| `components/src/dynamo/vllm/multimodal_handlers/processor_handler.py` | Processor implementation |
+| `components/src/dynamo/vllm/multimodal_handlers/encode_worker_handler.py` | Encode worker implementations (custom and vLLM-native) |
+| `components/src/dynamo/vllm/multimodal_handlers/worker_handler.py` | PD/Prefill/Decode worker implementation |

fern/pages/observability/README.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Dynamo Observability"
+---
+
+## Getting Started Quickly
+
+This is an example to get started quickly on a single machine.
+
+### Prerequisites
+
+Install these on your machine:
+
+- [Docker](https://docs.docker.com/get-docker/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+
+### Starting the Observability Stack
+
+Dynamo provides a Docker Compose-based observability stack that includes Prometheus, Grafana, Tempo, and various exporters for metrics, tracing, and visualization.
+
+From the Dynamo root directory:
+
+```bash
+# Start infrastructure (NATS, etcd)
+docker compose -f deploy/docker-compose.yml up -d
+
+# Start observability stack (Prometheus, Grafana, Tempo, DCGM GPU exporter, NATS exporter)
+docker compose -f deploy/docker-observability.yml up -d
+```
+
+For detailed setup instructions and configuration, see [Prometheus + Grafana Setup](prometheus-grafana.md).
+
+## Observability Documentations
+
+| Guide | Description | Environment Variables to Control |
+|-------|-------------|----------------------------------|
+| [Metrics](metrics.md) | Available metrics reference | `DYN_SYSTEM_PORT`† |
+| [Health Checks](health-checks.md) | Component health monitoring and readiness probes | `DYN_SYSTEM_PORT`†, `DYN_SYSTEM_STARTING_HEALTH_STATUS`, `DYN_SYSTEM_HEALTH_PATH`, `DYN_SYSTEM_LIVE_PATH`, `DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS` |
+| [Tracing](tracing.md) | Distributed tracing with OpenTelemetry and Tempo | `DYN_LOGGING_JSONL`†, `OTEL_EXPORT_ENABLED`†, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`†, `OTEL_SERVICE_NAME`† |
+| [Logging](logging.md) | Structured logging configuration | `DYN_LOGGING_JSONL`†, `DYN_LOG`, `DYN_LOG_USE_LOCAL_TZ`, `DYN_LOGGING_CONFIG_PATH`, `OTEL_SERVICE_NAME`†, `OTEL_EXPORT_ENABLED`†, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`† |
+
+**Variables marked with † are shared across multiple observability systems.**
+
+## Developer Guides
+
+| Guide | Description | Environment Variables to Control |
+|-------|-------------|----------------------------------|
+| [Metrics Developer Guide](metrics-developer-guide.md) | Creating custom metrics in Rust and Python | `DYN_SYSTEM_PORT`† |
+
+## Kubernetes
+
+For Kubernetes-specific setup and configuration, see [Kubernetes Metrics](../kubernetes/observability/metrics.md).
+
+---
+
+## Topology
+
+This provides:
+- **Prometheus** on `http://localhost:9090` - metrics collection and querying
+- **Grafana** on `http://localhost:3000` - visualization dashboards (username: `dynamo`, password: `dynamo`)
+- **Tempo** on `http://localhost:3200` - distributed tracing backend
+- **DCGM Exporter** on `http://localhost:9401/metrics` - GPU metrics
+- **NATS Exporter** on `http://localhost:7777/metrics` - NATS messaging metrics
+
+### Service Relationship Diagram
+```mermaid
+graph TD
+    BROWSER[Browser] -->|:3000| GRAFANA[Grafana :3000]
+    subgraph DockerComposeNetwork [Network inside Docker Compose]
+        NATS_PROM_EXP[nats-prom-exp :7777 /metrics] -->|:8222/varz| NATS_SERVER[nats-server :4222, :6222, :8222]
+        PROMETHEUS[Prometheus server :9090] -->|:2379/metrics| ETCD_SERVER[etcd-server :2379, :2380]
+        PROMETHEUS -->|:9401/metrics| DCGM_EXPORTER[dcgm-exporter :9401]
+        PROMETHEUS -->|:7777/metrics| NATS_PROM_EXP
+        PROMETHEUS -->|:8000/metrics| DYNAMOFE[Dynamo HTTP FE :8000]
+        PROMETHEUS -->|:8081/metrics| DYNAMOBACKEND[Dynamo backend :8081]
+        DYNAMOFE --> DYNAMOBACKEND
+        GRAFANA -->|:9090/query API| PROMETHEUS
+    end
+```
+
+The dcgm-exporter service in the Docker Compose network is configured to use port 9401 instead of the default port 9400. This adjustment is made to avoid port conflicts with other dcgm-exporter instances that may be running simultaneously. Such a configuration is typical in distributed systems like SLURM.
+
+### Configuration Files
+
+The following configuration files are located in the `deploy/observability/` directory:
+- [docker-compose.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/docker-compose.yml): Defines NATS and etcd services
+- [docker-observability.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/docker-observability.yml): Defines Prometheus, Grafana, Tempo, and exporters
+- [prometheus.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/prometheus.yml): Contains Prometheus scraping configuration
+- [grafana-datasources.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/grafana-datasources.yml): Contains Grafana datasource configuration
+- [grafana_dashboards/dashboard-providers.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/grafana_dashboards/dashboard-providers.yml): Contains Grafana dashboard provider configuration
+- [grafana_dashboards/dynamo.json](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/grafana_dashboards/dynamo.json): A general Dynamo Dashboard for both SW and HW metrics
+- [grafana_dashboards/dcgm-metrics.json](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/grafana_dashboards/dcgm-metrics.json): Contains Grafana dashboard configuration for DCGM GPU metrics
+- [grafana_dashboards/kvbm.json](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/grafana_dashboards/kvbm.json): Contains Grafana dashboard configuration for KVBM metrics
+

fern/pages/observability/health-checks.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Dynamo Health Checks"
+---
+
+## Overview
+
+Dynamo provides health check and liveness HTTP endpoints for each component which
+can be used to configure startup, liveness and readiness probes in
+orchestration frameworks such as Kubernetes.
+
+## Environment Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DYN_SYSTEM_PORT` | System status server port | `8081` | `9090` |
+| `DYN_SYSTEM_STARTING_HEALTH_STATUS` | Initial health status | `notready` | `ready`, `notready` |
+| `DYN_SYSTEM_HEALTH_PATH` | Custom health endpoint path | `/health` | `/custom/health` |
+| `DYN_SYSTEM_LIVE_PATH` | Custom liveness endpoint path | `/live` | `/custom/live` |
+| `DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS` | Endpoints required for ready state | none | `["generate"]` |
+| `DYN_HEALTH_CHECK_ENABLED` | Enable canary health checks | `false` (K8s: `true`) | `true`, `false` |
+| `DYN_CANARY_WAIT_TIME` | Seconds before sending canary health check | `10` | `5`, `30` |
+| `DYN_HEALTH_CHECK_REQUEST_TIMEOUT` | Health check request timeout in seconds | `3` | `5`, `10` |
+
+## Getting Started Quickly
+
+Enable health checks and query endpoints:
+
+```bash
+# Start your Dynamo components (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+python -m dynamo.frontend &
+
+# Enable system status server on port 8081
+DYN_SYSTEM_PORT=8081 python -m dynamo.vllm --model Qwen/Qwen3-0.6B --enforce-eager &
+```
+
+Check health status:
+
+```bash
+# Frontend health (port 8000)
+curl -s localhost:8000/health | jq
+
+# Worker health (port 8081)
+curl -s localhost:8081/health | jq
+```
+
+## Frontend Liveness Check
+
+The frontend liveness endpoint reports a status of `live` as long as
+the service is running.
+
+> **Note**: Frontend liveness doesn't depend on worker health or liveness only on the Frontend service itself.
+
+### Example Request
+
+```
+curl -s localhost:8080/live -q | jq
+```
+
+### Example Response
+
+```
+{
+  "message": "Service is live",
+  "status": "live"
+}
+```
+
+## Frontend Health Check
+
+The frontend health endpoint reports a status of `healthy` as long as
+the service is running.  Once workers have been registered, the
+`health` endpoint will also list registered endpoints and instances.
+
+> **Note**: Frontend liveness doesn't depend on worker health or liveness only on the Frontend service itself.
+
+### Example Request
+
+```
+curl -v localhost:8080/health -q | jq
+```
+
+### Example Response
+
+Before workers are registered:
+
+```
+HTTP/1.1 200 OK
+content-type: application/json
+content-length: 72
+date: Wed, 03 Sep 2025 13:31:44 GMT
+
+{
+  "instances": [],
+  "message": "No endpoints available",
+  "status": "unhealthy"
+}
+```
+
+After workers are registered:
+
+```
+HTTP/1.1 200 OK
+content-type: application/json
+content-length: 609
+date: Wed, 03 Sep 2025 13:32:03 GMT
+
+{
+  "endpoints": [
+    "dyn://dynamo.backend.generate"
+  ],
+  "instances": [
+    {
+      "component": "backend",
+      "endpoint": "clear_kv_blocks",
+      "instance_id": 7587888160958628000,
+      "namespace": "dynamo",
+      "transport": {
+        "nats_tcp": "dynamo_backend.clear_kv_blocks-694d98147d54be25"
+      }
+    },
+    {
+      "component": "backend",
+      "endpoint": "generate",
+      "instance_id": 7587888160958628000,
+      "namespace": "dynamo",
+      "transport": {
+        "nats_tcp": "dynamo_backend.generate-694d98147d54be25"
+      }
+    },
+    {
+      "component": "backend",
+      "endpoint": "load_metrics",
+      "instance_id": 7587888160958628000,
+      "namespace": "dynamo",
+      "transport": {
+        "nats_tcp": "dynamo_backend.load_metrics-694d98147d54be25"
+      }
+    }
+  ],
+  "status": "healthy"
+}
+```
+
+## Worker Liveness and Health Check
+
+Health checks for components other than the frontend are enabled
+selectively based on environment variables. If a health check for a
+component is enabled the starting status can be set along with the set
+of endpoints that are required to be served before the component is
+declared `ready`.
+
+Once all endpoints declared in `DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS`
+are served the component transitions to a `ready` state until the
+component is shutdown. The endpoints return HTTP status code of `HTTP/1.1 503 Service Unavailable`
+when initializing and HTTP status code `HTTP/1.1 200 OK` once ready.
+
+> **Note**: Both /live and /ready return the same information
+
+### Example Environment Setting
+
+```
+export DYN_SYSTEM_PORT=9090
+export DYN_SYSTEM_STARTING_HEALTH_STATUS="notready"
+export DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS="[\"generate\"]"
+```
+
+#### Example Request
+
+```
+curl -v localhost:9090/health | jq
+```
+
+#### Example Response
+Before endpoints are being served:
+
+```
+HTTP/1.1 503 Service Unavailable
+content-type: text/plain; charset=utf-8
+content-length: 96
+date: Wed, 03 Sep 2025 13:42:39 GMT
+
+{
+  "endpoints": {
+    "generate": "notready"
+  },
+  "status": "notready",
+  "uptime": {
+    "nanos": 313803539,
+    "secs": 12
+  }
+}
+```
+
+After endpoints are being served:
+
+```
+HTTP/1.1 200 OK
+content-type: text/plain; charset=utf-8
+content-length: 139
+date: Wed, 03 Sep 2025 13:42:45 GMT
+
+{
+  "endpoints": {
+    "clear_kv_blocks": "ready",
+    "generate": "ready",
+    "load_metrics": "ready"
+  },
+  "status": "ready",
+  "uptime": {
+    "nanos": 356504530,
+    "secs": 18
+  }
+}
+```
+
+## Canary Health Checks (Active Monitoring)
+
+In addition to the HTTP endpoints described above, Dynamo includes a **canary health check** system that actively monitors worker endpoints.
+
+### Overview
+
+The canary health check system:
+- **Monitors endpoint health** by sending periodic test requests to worker endpoints
+- **Only activates during idle periods** - if there's ongoing traffic, health checks are skipped to avoid overhead
+- **Automatically enabled in Kubernetes** deployments via the operator
+- **Disabled by default** in local/development environments
+
+### How It Works
+
+1. **Idle Detection**: After no activity on an endpoint for a configurable wait time (default: 10 seconds), a canary health check is triggered
+2. **Health Check Request**: A lightweight test request is sent to the endpoint with a minimal payload (generates 1 token)
+3. **Activity Resets Timer**: If normal requests arrive, the canary timer resets and no health check is sent
+4. **Timeout Handling**: If a health check doesn't respond within the timeout (default: 3 seconds), the endpoint is marked as unhealthy
+
+### Configuration
+
+#### In Kubernetes (Enabled by Default)
+
+Health checks are automatically enabled by the Dynamo operator. No additional configuration is required.
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeployment
+metadata:
+  name: my-deployment
+spec:
+  services:
+    VllmWorker:
+      componentType: worker
+      replicas: 2
+      # Health checks automatically enabled by operator
+```
+
+#### In Local/Development Environments (Disabled by Default)
+
+To enable health checks locally:
+
+```bash
+# Enable health checks
+export DYN_HEALTH_CHECK_ENABLED=true
+
+# Optional: Customize timing
+export DYN_CANARY_WAIT_TIME=5  # Wait 5 seconds before sending health check
+export DYN_HEALTH_CHECK_REQUEST_TIMEOUT=5  # 5 second timeout
+
+# Start worker
+python -m dynamo.vllm --model Qwen/Qwen3-0.6B
+```
+
+#### Configuration Options
+
+| Environment Variable | Description | Default | Notes |
+|---------------------|-------------|---------|-------|
+| `DYN_HEALTH_CHECK_ENABLED` | Enable/disable canary health checks | `false` (K8s: `true`) | Automatically set to `true` in K8s |
+| `DYN_CANARY_WAIT_TIME` | Seconds to wait (during idle) before sending health check | `10` | Lower values = more frequent checks |
+| `DYN_HEALTH_CHECK_REQUEST_TIMEOUT` | Max seconds to wait for health check response | `3` | Higher values = more tolerance for slow responses |
+
+### Health Check Payloads
+
+Each backend defines its own minimal health check payload:
+
+- **vLLM**: Single token generation with minimal sampling options
+- **TensorRT-LLM**: Single token with BOS token ID
+- **SGLang**: Single token generation request
+
+These payloads are designed to:
+- Complete quickly (< 100ms typically)
+- Minimize GPU overhead
+- Verify the full inference stack is working
+
+### Observing Health Checks
+
+When health checks are enabled, you'll see logs like:
+
+```
+INFO Health check manager started (canary_wait_time: 10s, request_timeout: 3s)
+INFO Spawned health check task for endpoint: generate
+INFO Canary timer expired for generate, sending health check
+INFO Health check successful for generate
+```
+
+If an endpoint fails:
+
+```
+WARN Health check timeout for generate
+ERROR Health check request failed for generate: connection refused
+```
+
+### When to Use Canary Health Checks
+
+**Enable in production (Kubernetes):**
+- ✅ Detect unhealthy workers before they affect user traffic
+- ✅ Enable faster failure detection and recovery
+- ✅ Monitor worker availability continuously
+
+**Disable in development:**
+- ✅ Reduce log noise during debugging
+- ✅ Avoid overhead when not needed
+- ✅ Simplify local testing
+
+### Troubleshooting
+
+**Health checks timing out:**
+- Increase `DYN_HEALTH_CHECK_REQUEST_TIMEOUT`
+- Check worker logs for errors
+- Verify network connectivity
+
+**Too many health check logs:**
+- Increase `DYN_CANARY_WAIT_TIME` to reduce frequency
+- Or disable with `DYN_HEALTH_CHECK_ENABLED=false` in dev
+
+**Health checks not running:**
+- Verify `DYN_HEALTH_CHECK_ENABLED=true` is set
+- Check that `DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS` includes the endpoint
+- Ensure the worker is serving the endpoint
+
+## Related Documentation
+
+- [Distributed Runtime Architecture](../design-docs/distributed-runtime.md)
+- [Dynamo Architecture Overview](../design-docs/architecture.md)
+- [Backend Guide](../development/backend-guide.md)

fern/pages/observability/logging.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Dynamo Logging"
+---
+
+## Overview
+
+Dynamo provides structured logging in both text as well as JSONL. When
+JSONL is enabled logs additionally contain `span` creation and exit
+events as well as support for `trace_id` and `span_id` fields for
+distributed tracing.
+
+## Environment Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DYN_LOGGING_JSONL` | Enable JSONL logging format | `false` | `true` |
+| `DYN_LOG` | Log levels per target `<default_level>,<module_path>=<level>,<module_path>=<level>` | `info` | `DYN_LOG=info,dynamo_runtime::system_status_server:trace` |
+| `DYN_LOG_USE_LOCAL_TZ` | Use local timezone for timestamps (default is UTC) | `false` | `true` |
+| `DYN_LOGGING_CONFIG_PATH` | Path to custom TOML logging configuration | none | `/path/to/config.toml` |
+| `OTEL_SERVICE_NAME` | Service name for trace and span information | `dynamo` | `dynamo-frontend` |
+| `OTEL_EXPORT_ENABLED` | Enable OTLP trace exporting | `false` | `true` |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | OTLP exporter endpoint | `http://localhost:4317` | `http://tempo:4317` |
+
+## Getting Started Quickly
+
+### Start Observability Stack
+
+For collecting and visualizing logs with Grafana Loki (Kubernetes), or viewing trace context in logs alongside Grafana Tempo, start the observability stack. See [Observability Getting Started](README.md#getting-started-quickly) for instructions.
+
+### Enable Structured Logging
+
+Enable structured JSONL logging:
+
+```bash
+export DYN_LOGGING_JSONL=true
+export DYN_LOG=debug
+
+# Start your Dynamo components (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+python -m dynamo.frontend &
+python -m dynamo.vllm --model Qwen/Qwen3-0.6B --enforce-eager &
+```
+
+Logs will be written to stderr in JSONL format with trace context.
+
+## Available Logging Levels
+
+| **Logging Levels (Least to Most Verbose)** | **Description**                                                                 |
+|-------------------------------------------|---------------------------------------------------------------------------------|
+| **ERROR**                                 | Critical errors (e.g., unrecoverable failures, resource exhaustion)              |
+| **WARN**                                  | Unexpected or degraded situations (e.g., retries, recoverable errors)           |
+| **INFO**                                  | Operational information (e.g., startup/shutdown, major events)                 |
+| **DEBUG**                                 | General debugging information (e.g., variable values, flow control)            |
+| **TRACE**                                 | Very low-level, detailed information (e.g., internal algorithm steps)           |
+
+## Example Readable Format
+
+Environment Setting:
+
+```
+export DYN_LOG="info,dynamo_runtime::system_status_server:trace"
+export DYN_LOGGING_JSONL="false"
+```
+
+Resulting Log format:
+
+```
+2025-09-02T15:50:01.770028Z  INFO main.init: VllmWorker for Qwen/Qwen3-0.6B has been initialized
+2025-09-02T15:50:01.770195Z  INFO main.init: Reading Events from tcp://127.0.0.1:21555
+2025-09-02T15:50:01.770265Z  INFO main.init: Getting engine runtime configuration metadata from vLLM engine...
+2025-09-02T15:50:01.770316Z  INFO main.get_engine_cache_info: Cache config values: {'num_gpu_blocks': 24064}
+2025-09-02T15:50:01.770358Z  INFO main.get_engine_cache_info: Scheduler config values: {'max_num_seqs': 256, 'max_num_batched_tokens': 2048}
+```
+
+## Example JSONL Format
+
+Environment Setting:
+
+```
+export DYN_LOG="info,dynamo_runtime::system_status_server:trace"
+export DYN_LOGGING_JSONL="true"
+```
+
+Resulting Log format:
+
+```
+{"time":"2025-09-02T15:53:31.943377Z","level":"INFO","target":"log","message":"VllmWorker for Qwen/Qwen3-0.6B has been initialized","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":191,"log.target":"main.init"}
+{"time":"2025-09-02T15:53:31.943550Z","level":"INFO","target":"log","message":"Reading Events from tcp://127.0.0.1:26771","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":212,"log.target":"main.init"}
+{"time":"2025-09-02T15:53:31.943636Z","level":"INFO","target":"log","message":"Getting engine runtime configuration metadata from vLLM engine...","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":220,"log.target":"main.init"}
+{"time":"2025-09-02T15:53:31.943701Z","level":"INFO","target":"log","message":"Cache config values: {'num_gpu_blocks': 24064}","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":267,"log.target":"main.get_engine_cache_info"}
+{"time":"2025-09-02T15:53:31.943747Z","level":"INFO","target":"log","message":"Scheduler config values: {'max_num_seqs': 256, 'max_num_batched_tokens': 2048}","log.file":"/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/vllm/main.py","log.line":268,"log.target":"main.get_engine_cache_info"}
+```
+
+## Logging of Trace and Span IDs
+
+When `DYN_LOGGING_JSONL` is enabled, all logs include `trace_id` and `span_id` fields, and spans are automatically created for requests. This is useful for short debugging sessions where you want to examine trace context in logs without setting up a full tracing backend and for correlating log messages with traces.
+
+The trace and span information uses the OpenTelemetry format and libraries, which means the IDs are compatible with OpenTelemetry-based tracing backends like Tempo or Jaeger if you later choose to enable trace export.
+
+**Note:** This section has overlap with [Distributed Tracing with Tempo](tracing.md). For trace visualization in Grafana Tempo and persistent trace analysis, see [Distributed Tracing with Tempo](tracing.md).
+
+### Configuration for Logging
+
+To see trace information in logs:
+
+```bash
+export DYN_LOGGING_JSONL=true
+export DYN_LOG=debug  # Set to debug to see detailed trace logs
+
+# Start your Dynamo components (e.g., frontend and worker) (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+python -m dynamo.frontend &
+python -m dynamo.vllm --model Qwen/Qwen3-0.6B --enforce-eager &
+```
+
+This enables JSONL logging with `trace_id` and `span_id` fields. Traces appear in logs but are not exported to any backend.
+
+### Example Request
+
+Send a request to generate logs with trace context:
+
+```bash
+curl -H 'Content-Type: application/json' \
+-H 'x-request-id: test-trace-001' \
+-d '{
+  "model": "Qwen/Qwen3-0.6B",
+  "max_completion_tokens": 100,
+  "messages": [
+    {"role": "user", "content": "What is the capital of France?"}
+  ]
+}' \
+http://localhost:8000/v1/chat/completions
+```
+
+Check the logs (stderr) for JSONL output containing `trace_id`, `span_id`, and `x_request_id` fields.
+
+## Trace and Span Information in Logs
+
+This section shows how trace and span information appears in JSONL logs. These logs can be used to understand request flows even without a trace visualization backend.
+
+### Example Disaggregated Trace in Grafana
+
+When viewing the corresponding trace in Grafana, you should be able to see something like the following:
+
+![Disaggregated Trace Example](../../assets/img/grafana-disagg-trace.png)
+
+### Trace Overview
+
+| Attribute | Value |
+|-----------|-------|
+| **Trace ID** | b672ccf48683b392891c5cb4163d4b51 |
+| **Start Time** | 2025-10-31 13:52:10.706 |
+| **Duration** | 4.04s |
+| **Request** | `POST /v1/chat/completions` |
+
+### Root Span (Frontend): `http-request`
+
+| Attribute | Value |
+|-----------|-------|
+| **Service** | frontend |
+| **Span ID** | 5c20cc08e6afb2b7 |
+| **Duration** | 4.04s |
+| **Start Time** | 13:52:10.706 |
+| **Status** | unset |
+| **Method** | POST |
+| **URI** | `/v1/chat/completions` |
+| **HTTP Version** | HTTP/1.1 |
+| **Parent ID** | (none) |
+| **Child Count** | 2 |
+| **Busy Time** | 18,101,350 ns (18.10ms) |
+| **Idle Time** | 4,022,100,356 ns (4.02s) |
+
+### Child Span (Prefill): `handle_payload`
+
+| Attribute | Value |
+|-----------|-------|
+| **Service** | prefill |
+| **Duration** | 39.65ms |
+| **Start Time** | 13:52:10.707 |
+| **Status** | unset |
+| **Component** | prefill |
+| **Endpoint** | generate |
+| **Namespace** | vllm-disagg |
+| **Instance ID** | 3866790875219207267 |
+| **Trace ID** | b672ccf48683b392891c5cb4163d4b51 |
+| **Parent ID** | 5c20cc08e6afb2b7 |
+| **Busy Time** | 613,633 ns (0.61ms) |
+| **Idle Time** | 36,340,242 ns (36.34ms) |
+
+### Child Span (Decode): `handle_payload`
+
+| Attribute | Value |
+|-----------|-------|
+| **Service** | decode |
+| **Duration** | 4s |
+| **Start Time** | 13:52:10.745 |
+| **Status** | unset |
+| **Component** | backend |
+| **Endpoint** | generate |
+| **Namespace** | vllm-disagg |
+| **Instance ID** | 3866790875219207263 |
+| **Trace ID** | b672ccf48683b392891c5cb4163d4b51 |
+| **Parent ID** | 5c20cc08e6afb2b7 |
+| **Busy Time** | 3,795,258 ns (3.79ms) |
+| **Idle Time** | 3,996,532,471 ns (3.99s) |
+
+### Frontend Logs with Trace Context
+
+The following shows the JSONL logs from the frontend service for the same request. Note the `trace_id` field (`b672ccf48683b392891c5cb4163d4b51`) that correlates all logs for this request, and the `span_id` field that identifies individual operations:
+
+```
+{"time":"2025-10-31T20:52:07.707164Z","level":"INFO","file":"/opt/dynamo/lib/runtime/src/logging.rs","line":806,"target":"dynamo_runtime::logging","message":"OTLP export enabled","endpoint":"http://tempo.tm.svc.cluster.local:4317","service":"frontend"}
+{"time":"2025-10-31T20:52:10.707164Z","level":"DEBUG","file":"/opt/dynamo/lib/runtime/src/pipeline/network/tcp/server.rs","line":230,"target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream on 10.0.4.65:41959","method":"POST","span_id":"5c20cc08e6afb2b7","span_name":"http-request","trace_id":"b672ccf48683b392891c5cb4163d4b51","uri":"/v1/chat/completions","version":"HTTP/1.1"}
+{"time":"2025-10-31T20:52:10.745264Z","level":"DEBUG","file":"/opt/dynamo/lib/llm/src/kv_router/prefill_router.rs","line":232,"target":"dynamo_llm::kv_router::prefill_router","message":"Prefill succeeded, using disaggregated params for decode","method":"POST","span_id":"5c20cc08e6afb2b7","span_name":"http-request","trace_id":"b672ccf48683b392891c5cb4163d4b51","uri":"/v1/chat/completions","version":"HTTP/1.1"}
+{"time":"2025-10-31T20:52:10.745545Z","level":"DEBUG","file":"/opt/dynamo/lib/runtime/src/pipeline/network/tcp/server.rs","line":230,"target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream on 10.0.4.65:41959","method":"POST","span_id":"5c20cc08e6afb2b7","span_name":"http-request","trace_id":"b672ccf48683b392891c5cb4163d4b51","uri":"/v1/chat/completions","version":"HTTP/1.1"}
+```
+
+## Custom Request IDs in Logs
+
+You can provide a custom request ID using the `x-request-id` header. This ID will be attached to all spans and logs for that request, making it easier to correlate traces with application-level request tracking.
+
+### Example Request with Custom Request ID
+
+```sh
+curl -X POST http://localhost:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -H 'x-request-id: 8372eac7-5f43-4d76-beca-0a94cfb311d0' \
+  -d '{
+    "model": "Qwen/Qwen3-0.6B",
+    "messages": [
+      {
+        "role": "user",
+        "content": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
+      }
+    ],
+    "stream": false,
+    "max_tokens": 1000
+  }'
+```
+
+All spans and logs for this request will include the `x_request_id` attribute with value `8372eac7-5f43-4d76-beca-0a94cfb311d0`.
+
+### Frontend Logs with Custom Request ID
+
+Notice how the `x_request_id` field appears in all log entries, alongside the `trace_id` (`80196f3e3a6fdf06d23bb9ada3788518`) and `span_id`:
+
+```
+{"time":"2025-10-31T21:06:45.397194Z","level":"DEBUG","file":"/opt/dynamo/lib/runtime/src/pipeline/network/tcp/server.rs","line":230,"target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream on 10.0.4.65:41959","method":"POST","span_id":"f7e487a9d2a6bf38","span_name":"http-request","trace_id":"80196f3e3a6fdf06d23bb9ada3788518","uri":"/v1/chat/completions","version":"HTTP/1.1","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
+{"time":"2025-10-31T21:06:45.418584Z","level":"DEBUG","file":"/opt/dynamo/lib/llm/src/kv_router/prefill_router.rs","line":232,"target":"dynamo_llm::kv_router::prefill_router","message":"Prefill succeeded, using disaggregated params for decode","method":"POST","span_id":"f7e487a9d2a6bf38","span_name":"http-request","trace_id":"80196f3e3a6fdf06d23bb9ada3788518","uri":"/v1/chat/completions","version":"HTTP/1.1","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
+{"time":"2025-10-31T21:06:45.418854Z","level":"DEBUG","file":"/opt/dynamo/lib/runtime/src/pipeline/network/tcp/server.rs","line":230,"target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream on 10.0.4.65:41959","method":"POST","span_id":"f7e487a9d2a6bf38","span_name":"http-request","trace_id":"80196f3e3a6fdf06d23bb9ada3788518","uri":"/v1/chat/completions","version":"HTTP/1.1","x_request_id":"8372eac7-5f43-4d76-beca-0a94cfb311d0"}
+```
+
+
+
+## Related Documentation
+
+- [Distributed Runtime Architecture](../design-docs/distributed-runtime.md)
+- [Dynamo Architecture Overview](../design-docs/architecture.md)
+- [Backend Guide](../development/backend-guide.md)
+- [Log Aggregation in Kubernetes](../kubernetes/observability/logging.md)

fern/pages/observability/metrics-developer-guide.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Metrics Developer Guide"
+---
+
+This guide explains how to create and use custom metrics in Dynamo components using the Dynamo metrics API.
+
+## Metrics Exposure
+
+All metrics created via the Dynamo metrics API are automatically exposed on the `/metrics` HTTP endpoint in Prometheus Exposition Format text when the following environment variable is set:
+
+- `DYN_SYSTEM_PORT=<port>` - Port for the metrics endpoint (set to positive value to enable, default: `-1` disabled)
+
+Example:
+```bash
+DYN_SYSTEM_PORT=8081 python -m dynamo.vllm --model <model>
+```
+
+Prometheus Exposition Format text metrics will be available at: `http://localhost:8081/metrics`
+
+## Metric Name Constants
+
+The [prometheus_names.rs](https://github.com/ai-dynamo/dynamo/tree/main/lib/runtime/src/metrics/prometheus_names.rs) module provides centralized metric name constants and sanitization functions to ensure consistency across all Dynamo components.
+
+---
+
+## Metrics API in Rust
+
+The metrics API is accessible through the `.metrics()` method on runtime, namespace, component, and endpoint objects. See [Runtime Hierarchy](metrics.md#runtime-hierarchy) for details on the hierarchical structure.
+
+### Available Methods
+
+- `.metrics().create_counter()`: Create a counter metric
+- `.metrics().create_gauge()`: Create a gauge metric
+- `.metrics().create_histogram()`: Create a histogram metric
+- `.metrics().create_countervec()`: Create a counter with labels
+- `.metrics().create_gaugevec()`: Create a gauge with labels
+- `.metrics().create_histogramvec()`: Create a histogram with labels
+
+### Creating Metrics
+
+```rust
+use dynamo_runtime::DistributedRuntime;
+
+let runtime = DistributedRuntime::new()?;
+let endpoint = runtime.namespace("my_namespace").component("my_component").endpoint("my_endpoint");
+
+// Simple metrics
+let requests_total = endpoint.metrics().create_counter(
+    "requests_total",
+    "Total requests",
+    &[]
+)?;
+
+let active_connections = endpoint.metrics().create_gauge(
+    "active_connections",
+    "Active connections",
+    &[]
+)?;
+
+let latency = endpoint.metrics().create_histogram(
+    "latency_seconds",
+    "Request latency",
+    &[],
+    Some(vec![0.001, 0.01, 0.1, 1.0, 10.0])
+)?;
+```
+
+### Using Metrics
+
+```rust
+// Counters
+requests_total.inc();
+
+// Gauges
+active_connections.set(42.0);
+active_connections.inc();
+active_connections.dec();
+
+// Histograms
+latency.observe(0.023);  // 23ms
+```
+
+### Vector Metrics with Labels
+
+```rust
+// Create vector metrics with label names
+let requests_by_model = endpoint.metrics().create_countervec(
+    "requests_by_model",
+    "Requests by model",
+    &["model_type", "model_size"],
+    &[]
+)?;
+
+let memory_by_gpu = endpoint.metrics().create_gaugevec(
+    "gpu_memory_bytes",
+    "GPU memory by device",
+    &["gpu_id", "memory_type"],
+    &[]
+)?;
+
+// Use with specific label values
+requests_by_model.with_label_values(&["llama", "7b"]).inc();
+memory_by_gpu.with_label_values(&["0", "allocated"]).set(8192.0);
+```
+
+### Advanced Features
+
+**Custom histogram buckets:**
+```rust
+let latency = endpoint.metrics().create_histogram(
+    "latency_seconds",
+    "Request latency",
+    &[],
+    Some(vec![0.001, 0.01, 0.1, 1.0, 10.0])
+)?;
+```
+
+**Constant labels:**
+```rust
+let counter = endpoint.metrics().create_counter(
+    "requests_total",
+    "Total requests",
+    &[("region", "us-west"), ("env", "prod")]
+)?;
+```
+
+---
+
+## Metrics API in Python
+
+Python components can create and manage Prometheus metrics using the same metrics API through Python bindings.
+
+### Available Methods
+
+- `endpoint.metrics.create_counter()` / `create_intcounter()`: Create a counter metric
+- `endpoint.metrics.create_gauge()` / `create_intgauge()`: Create a gauge metric
+- `endpoint.metrics.create_histogram()`: Create a histogram metric
+- `endpoint.metrics.create_countervec()` / `create_intcountervec()`: Create a counter with labels
+- `endpoint.metrics.create_gaugevec()` / `create_intgaugevec()`: Create a gauge with labels
+- `endpoint.metrics.create_histogramvec()`: Create a histogram with labels
+
+All metrics are imported from `dynamo.prometheus_metrics`.
+
+### Creating Metrics
+
+```python
+from dynamo.runtime import DistributedRuntime
+
+drt = DistributedRuntime()
+endpoint = drt.namespace("my_namespace").component("my_component").endpoint("my_endpoint")
+
+# Simple metrics
+requests_total = endpoint.metrics.create_intcounter(
+    "requests_total",
+    "Total requests"
+)
+
+active_connections = endpoint.metrics.create_intgauge(
+    "active_connections",
+    "Active connections"
+)
+
+latency = endpoint.metrics.create_histogram(
+    "latency_seconds",
+    "Request latency",
+    buckets=[0.001, 0.01, 0.1, 1.0, 10.0]
+)
+```
+
+### Using Metrics
+
+```python
+# Counters
+requests_total.inc()
+requests_total.inc_by(5)
+
+# Gauges
+active_connections.set(42)
+active_connections.inc()
+active_connections.dec()
+
+# Histograms
+latency.observe(0.023)  # 23ms
+```
+
+### Vector Metrics with Labels
+
+```python
+# Create vector metrics with label names
+requests_by_model = endpoint.metrics.create_intcountervec(
+    "requests_by_model",
+    "Requests by model",
+    ["model_type", "model_size"]
+)
+
+memory_by_gpu = endpoint.metrics.create_intgaugevec(
+    "gpu_memory_bytes",
+    "GPU memory by device",
+    ["gpu_id", "memory_type"]
+)
+
+# Use with specific label values
+requests_by_model.inc({"model_type": "llama", "model_size": "7b"})
+memory_by_gpu.set(8192, {"gpu_id": "0", "memory_type": "allocated"})
+```
+
+### Advanced Features
+
+**Constant labels:**
+```python
+counter = endpoint.metrics.create_intcounter(
+    "requests_total",
+    "Total requests",
+    [("region", "us-west"), ("env", "prod")]
+)
+```
+
+**Metric introspection:**
+```python
+print(counter.name())            # "my_namespace_my_component_my_endpoint_requests_total"
+print(counter.const_labels())    # {"dynamo_namespace": "my_namespace", ...}
+print(gauge_vec.variable_labels())  # ["model_type", "model_size"]
+```
+
+**Update patterns:**
+
+Background thread updates:
+```python
+import threading
+import time
+
+def update_loop():
+    while True:
+        active_connections.set(compute_current_connections())
+        time.sleep(2)
+
+threading.Thread(target=update_loop, daemon=True).start()
+```
+
+Callback-based updates (called before each `/metrics` scrape):
+```python
+def update_metrics():
+    active_connections.set(compute_current_connections())
+
+endpoint.metrics.register_callback(update_metrics)
+```
+
+### Examples
+
+Example scripts: [lib/bindings/python/examples/metrics/](https://github.com/ai-dynamo/dynamo/tree/main/lib/bindings/python/examples/metrics/)
+
+```bash
+cd ~/dynamo/lib/bindings/python/examples/metrics
+DYN_SYSTEM_PORT=8081 ./server_with_loop.py
+DYN_SYSTEM_PORT=8081 ./server_with_callback.py
+```
+
+---
+
+## Related Documentation
+
+- [Metrics Overview](metrics.md)
+- [Prometheus and Grafana Setup](prometheus-grafana.md)
+- [Distributed Runtime Architecture](../design-docs/distributed-runtime.md)
+- [Python Metrics Examples](https://github.com/ai-dynamo/dynamo/tree/main/lib/bindings/python/examples/metrics/)
+

fern/pages/observability/metrics.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Dynamo Metrics"
+---
+
+## Overview
+
+Dynamo provides built-in metrics capabilities through the Dynamo metrics API, which is automatically available whenever you use the `DistributedRuntime` framework. This document serves as a reference for all available metrics in Dynamo.
+
+**For visualization setup instructions**, see the [Prometheus and Grafana Setup Guide](prometheus-grafana.md).
+
+**For creating custom metrics**, see the [Metrics Developer Guide](metrics-developer-guide.md).
+
+## Environment Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DYN_SYSTEM_PORT` | Backend component metrics/health port | `-1` (disabled) | `8081` |
+| `DYN_HTTP_PORT` | Frontend HTTP port (also configurable via `--http-port` flag) | `8000` | `8000` |
+
+## Getting Started Quickly
+
+This is a single machine example.
+
+### Start Observability Stack
+
+For visualizing metrics with Prometheus and Grafana, start the observability stack. See [Observability Getting Started](README.md#getting-started-quickly) for instructions.
+
+
+### Launch Dynamo Components
+
+Launch a frontend and vLLM backend to test metrics:
+
+```bash
+# Start frontend (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+$ python -m dynamo.frontend
+
+# Enable backend worker's system metrics on port 8081
+$ DYN_SYSTEM_PORT=8081 python -m dynamo.vllm --model Qwen/Qwen3-0.6B  \
+   --enforce-eager --no-enable-prefix-caching --max-num-seqs 3
+```
+
+Wait for the vLLM worker to start, then send requests and check metrics:
+
+```bash
+# Send a request
+curl -H 'Content-Type: application/json' \
+-d '{
+  "model": "Qwen/Qwen3-0.6B",
+  "max_completion_tokens": 100,
+  "messages": [{"role": "user", "content": "Hello"}]
+}' \
+http://localhost:8000/v1/chat/completions
+
+# Check metrics from the backend worker
+curl -s localhost:8081/metrics | grep dynamo_component
+```
+
+## Exposed Metrics
+
+Dynamo exposes metrics in Prometheus Exposition Format text at the `/metrics` HTTP endpoint. All Dynamo-generated metrics use the `dynamo_*` prefix and include labels (`dynamo_namespace`, `dynamo_component`, `dynamo_endpoint`) to identify the source component.
+
+**Example Prometheus Exposition Format text:**
+
+```
+# HELP dynamo_component_requests_total Total requests processed
+# TYPE dynamo_component_requests_total counter
+dynamo_component_requests_total{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate"} 42
+
+# HELP dynamo_component_request_duration_seconds Request processing time
+# TYPE dynamo_component_request_duration_seconds histogram
+dynamo_component_request_duration_seconds_bucket{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate",le="0.005"} 10
+dynamo_component_request_duration_seconds_bucket{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate",le="0.01"} 15
+dynamo_component_request_duration_seconds_bucket{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate",le="+Inf"} 42
+dynamo_component_request_duration_seconds_sum{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate"} 2.5
+dynamo_component_request_duration_seconds_count{dynamo_namespace="default",dynamo_component="worker",dynamo_endpoint="generate"} 42
+```
+
+### Metric Categories
+
+Dynamo exposes several categories of metrics:
+
+- **Frontend Metrics** (`dynamo_frontend_*`) - Request handling, token processing, and latency measurements
+- **Component Metrics** (`dynamo_component_*`) - Request counts, processing times, byte transfers, and system uptime
+- **Specialized Component Metrics** (e.g., `dynamo_preprocessor_*`) - Component-specific metrics
+- **Engine Metrics** (Pass-through) - Backend engines expose their own metrics: [vLLM](../backends/vllm/prometheus.md) (`vllm:*`), [SGLang](../backends/sglang/prometheus.md) (`sglang:*`), [TensorRT-LLM](../backends/trtllm/prometheus.md) (`trtllm_*`)
+
+## Runtime Hierarchy
+
+The Dynamo metrics API is available on `DistributedRuntime`, `Namespace`, `Component`, and `Endpoint`, providing a hierarchical approach to metric collection that matches Dynamo's distributed architecture:
+
+- `DistributedRuntime`: Global metrics across the entire runtime
+- `Namespace`: Metrics scoped to a specific dynamo_namespace
+- `Component`: Metrics for a specific dynamo_component within a namespace
+- `Endpoint`: Metrics for individual dynamo_endpoint within a component
+
+This hierarchical structure allows you to create metrics at the appropriate level of granularity for your monitoring needs.
+
+## Available Metrics
+
+### Backend Component Metrics
+
+**Backend workers** (`python -m dynamo.vllm`, `python -m dynamo.sglang`, etc.) expose `dynamo_component_*` metrics on port 8081 by default (configurable via `DYN_SYSTEM_PORT`).
+
+The core Dynamo backend system automatically exposes metrics on the system status port (default: 8081, configurable via `DYN_SYSTEM_PORT`) at the `/metrics` endpoint with the `dynamo_component_*` prefix for all components that use the `DistributedRuntime` framework:
+
+- `dynamo_component_inflight_requests`: Requests currently being processed (gauge)
+- `dynamo_component_request_bytes_total`: Total bytes received in requests (counter)
+- `dynamo_component_request_duration_seconds`: Request processing time (histogram)
+- `dynamo_component_requests_total`: Total requests processed (counter)
+- `dynamo_component_response_bytes_total`: Total bytes sent in responses (counter)
+- `dynamo_component_uptime_seconds`: DistributedRuntime uptime (gauge)
+
+**Access backend component metrics:**
+```bash
+# Default port 8081
+curl http://localhost:8081/metrics
+
+# Or with custom port
+DYN_SYSTEM_PORT=8081 python -m dynamo.vllm --model <model>
+curl http://localhost:8081/metrics
+```
+
+### KV Router Statistics (kvstats)
+
+KV router statistics are automatically exposed by LLM workers and KV router components on the backend system status port (port 8081) with the `dynamo_component_kvstats_*` prefix. These metrics provide insights into GPU memory usage and cache efficiency:
+
+- `dynamo_component_kvstats_active_blocks`: Number of active KV cache blocks currently in use (gauge)
+- `dynamo_component_kvstats_total_blocks`: Total number of KV cache blocks available (gauge)
+- `dynamo_component_kvstats_gpu_cache_usage_percent`: GPU cache usage as a percentage (0.0-1.0) (gauge)
+- `dynamo_component_kvstats_gpu_prefix_cache_hit_rate`: GPU prefix cache hit rate as a percentage (0.0-1.0) (gauge)
+
+These metrics are published by:
+- **LLM Workers**: vLLM and TRT-LLM backends publish these metrics through their respective publishers
+- **KV Router**: The KV router component aggregates and exposes these metrics for load balancing decisions
+
+### Specialized Component Metrics
+
+Some components expose additional metrics specific to their functionality:
+
+- `dynamo_preprocessor_*`: Metrics specific to preprocessor components
+
+### Frontend Metrics
+
+**Important:** The frontend and backend workers are separate components that expose metrics on different ports. See [Backend Component Metrics](#backend-component-metrics) for backend metrics.
+
+The Dynamo HTTP Frontend (`python -m dynamo.frontend`) exposes `dynamo_frontend_*` metrics on port 8000 by default (configurable via `--http-port` or `DYN_HTTP_PORT`) at the `/metrics` endpoint. Most metrics include `model` labels containing the model name:
+
+- `dynamo_frontend_inflight_requests`: Inflight requests (gauge)
+- `dynamo_frontend_queued_requests`: Number of requests in HTTP processing queue (gauge)
+- `dynamo_frontend_disconnected_clients`: Number of disconnected clients (gauge)
+- `dynamo_frontend_input_sequence_tokens`: Input sequence length (histogram)
+- `dynamo_frontend_cached_tokens`: Number of cached tokens (prefix cache hits) per request (histogram)
+- `dynamo_frontend_inter_token_latency_seconds`: Inter-token latency (histogram)
+- `dynamo_frontend_output_sequence_tokens`: Output sequence length (histogram)
+- `dynamo_frontend_output_tokens_total`: Total number of output tokens generated (counter)
+- `dynamo_frontend_request_duration_seconds`: LLM request duration (histogram)
+- `dynamo_frontend_requests_total`: Total LLM requests (counter)
+- `dynamo_frontend_time_to_first_token_seconds`: Time to first token (histogram)
+- `dynamo_frontend_model_migration_total`: Total number of request migrations due to worker unavailability (counter, labels: `model`, `migration_type`)
+
+**Access frontend metrics:**
+```bash
+curl http://localhost:8000/metrics
+```
+
+**Note**: The `dynamo_frontend_inflight_requests` metric tracks requests from HTTP handler start until the complete response is finished, while `dynamo_frontend_queued_requests` tracks requests from HTTP handler start until first token generation begins (including prefill time). HTTP queue time is a subset of inflight time.
+
+#### Model Configuration Metrics
+
+The frontend also exposes model configuration metrics (on port 8000 `/metrics` endpoint) with the `dynamo_frontend_model_*` prefix. These metrics are populated from the worker backend registration service when workers register with the system. All model configuration metrics include a `model` label.
+
+**Runtime Config Metrics (from ModelRuntimeConfig):**
+These metrics come from the runtime configuration provided by worker backends during registration.
+
+- `dynamo_frontend_model_total_kv_blocks`: Total KV blocks available for a worker serving the model (gauge)
+- `dynamo_frontend_model_max_num_seqs`: Maximum number of sequences for a worker serving the model (gauge)
+- `dynamo_frontend_model_max_num_batched_tokens`: Maximum number of batched tokens for a worker serving the model (gauge)
+
+**MDC Metrics (from ModelDeploymentCard):**
+These metrics come from the Model Deployment Card information provided by worker backends during registration. Note that when multiple worker instances register with the same model name, only the first instance's configuration metrics (runtime config and MDC metrics) will be populated. Subsequent instances with duplicate model names will be skipped for configuration metric updates.
+
+- `dynamo_frontend_model_context_length`: Maximum context length for a worker serving the model (gauge)
+- `dynamo_frontend_model_kv_cache_block_size`: KV cache block size for a worker serving the model (gauge)
+- `dynamo_frontend_model_migration_limit`: Request migration limit for a worker serving the model (gauge)
+
+### Request Processing Flow
+
+This section explains the distinction between two key metrics used to track request processing:
+
+1. **Inflight**: Tracks requests from HTTP handler start until the complete response is finished
+2. **HTTP Queue**: Tracks requests from HTTP handler start until first token generation begins (including prefill time)
+
+**Example Request Flow:**
+```
+curl -s localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
+  "model": "Qwen/Qwen3-0.6B",
+  "prompt": "Hello let's talk about LLMs",
+  "stream": false,
+  "max_tokens": 1000
+}'
+```
+
+**Timeline:**
+```
+Timeline:    0, 1, ...
+Client ────> Frontend:8000 ────────────────────> Dynamo component/backend (vLLM, SGLang, TRT)
+             │request start                     │received                              │
+             |                                  |                                      |
+             │                                  ├──> start prefill ──> first token ──> |last token
+             │                                  │     (not impl)       |               |
+             ├─────actual HTTP queue¹ ──────────┘                      │               |
+             │                                                         │               │
+             ├─────implemented HTTP queue ─────────────────────────────┘               |
+             │                                                                         │
+             └─────────────────────────────────── Inflight ────────────────────────────┘
+```
+
+**Concurrency Example:**
+Suppose the backend allows 3 concurrent requests and there are 10 clients continuously hitting the frontend:
+- All 10 requests will be counted as inflight (from start until complete response)
+- 7 requests will be in HTTP queue most of the time
+- 3 requests will be actively processed (between first token and last token)
+
+**Key Differences:**
+- **Inflight**: Measures total request lifetime including processing time
+- **HTTP Queue**: Measures queuing time before processing begins (including prefill time)
+- **HTTP Queue ≤ Inflight** (HTTP queue is a subset of inflight time)
+
+## Related Documentation
+
+- [Distributed Runtime Architecture](../design-docs/distributed-runtime.md)
+- [Dynamo Architecture Overview](../design-docs/architecture.md)
+- [Backend Guide](../development/backend-guide.md)

fern/pages/observability/prometheus-grafana.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Metrics Visualization with Prometheus and Grafana"
+---
+
+## Overview
+
+This guide shows how to set up Prometheus and Grafana for visualizing Dynamo metrics on a single machine for demo purposes.
+
+![Grafana Dynamo Dashboard](../../assets/img/grafana-dynamo-composite.png)
+
+**Components:**
+- **Prometheus Server** - Collects and stores metrics from Dynamo services
+- **Grafana** - Provides dashboards by querying the Prometheus Server
+
+**For metrics reference**, see [Metrics Documentation](metrics.md).
+
+## Environment Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DYN_SYSTEM_PORT` | System metrics/health port | `-1` (disabled) | `8081` |
+
+## Getting Started Quickly
+
+This is a single machine example.
+
+### Start the Observability Stack
+
+Start the observability stack (Prometheus, Grafana, Tempo, exporters). See [Observability Getting Started](README.md#getting-started-quickly) for instructions and prerequisites.
+
+### Start Dynamo Components
+
+Start frontend and worker (a simple single GPU example):
+
+```bash
+# Start frontend (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+python -m dynamo.frontend &
+
+# Start vLLM worker with metrics enabled on port 8081
+DYN_SYSTEM_PORT=8081 python -m dynamo.vllm --model Qwen/Qwen3-0.6B --enforce-eager
+```
+
+After the workers are running, send a few test requests to populate metrics in the system:
+
+```bash
+curl -X POST http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen/Qwen3-0.6B",
+    "messages": [{"role": "user", "content": "Hello"}],
+    "max_completion_tokens": 100
+  }'
+```
+
+After sending a few requests, the Prometheus Exposition Format text metrics are available at:
+- Frontend: `http://localhost:8000/metrics`
+- Backend worker: `http://localhost:8081/metrics`
+
+### Access Web Interfaces
+
+Once Dynamo components are running:
+
+1. Open **Grafana** at `http://localhost:3000` (username: `dynamo`, password: `dynamo`)
+2. Click on **Dashboards** in the left sidebar
+3. Select **Dynamo Dashboard** to view metrics and traces
+
+Other interfaces:
+- **Prometheus**: `http://localhost:9090`
+- **Tempo** (tracing): Accessible through Grafana's Explore view. See [Tracing Guide](tracing.md) for details.
+
+**Note:** If accessing from another machine, replace `localhost` with the machine's hostname or IP address, and ensure firewall rules allow access to these ports (3000, 9090).
+
+---
+
+## Configuration
+
+### Prometheus
+
+The Prometheus configuration is specified in [prometheus.yml](https://github.com/ai-dynamo/dynamo/tree/main/deploy/observability/prometheus.yml). This file is set up to collect metrics from the metrics aggregation service endpoint.
+
+Please be aware that you might need to modify the target settings to align with your specific host configuration and network environment.
+
+After making changes to prometheus.yml, restart the Prometheus service. See [Observability Getting Started](README.md#getting-started-quickly) for Docker Compose commands.
+
+### Grafana
+
+Grafana is pre-configured with:
+- Prometheus datasource
+- Sample dashboard for visualizing service metrics
+
+### Troubleshooting
+
+1. Verify services are running using `docker compose ps`
+
+2. Check logs using `docker compose logs`
+
+3. Check Prometheus targets at `http://localhost:9090/targets` to verify metric collection.
+
+4. If you encounter issues with stale data or configuration, stop services and wipe volumes using `docker compose down -v` then restart.
+
+  **Note:** The `-v` flag removes named volumes (grafana-data, tempo-data), which will reset dashboards and stored metrics.
+
+For specific Docker Compose commands, see [Observability Getting Started](README.md#getting-started-quickly).
+
+## Developer Guide
+
+For detailed information on creating custom metrics in Dynamo components, see:
+
+- [Metrics Developer Guide](metrics-developer-guide.md)

fern/pages/observability/tracing.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Distributed Tracing with Tempo"
+---
+
+## Overview
+
+Dynamo supports OpenTelemetry-based distributed tracing for visualizing request flows across Frontend and Worker components. Traces are exported to Tempo via OTLP (OpenTelemetry Protocol) and visualized in Grafana.
+
+**Requirements:** Set `DYN_LOGGING_JSONL=true` and `OTEL_EXPORT_ENABLED=true` to export traces to Tempo.
+
+This guide covers single GPU demo setup using Docker Compose. For Kubernetes deployments, see [Kubernetes Deployment](#kubernetes-deployment).
+
+**Note:** This section has overlap with [Logging of OpenTelemetry Tracing](logging.md) since OpenTelemetry has aspects of both logging and tracing. The tracing approach documented here is for persistent trace visualization and analysis. For short debugging sessions examining trace context directly in logs, see the [Logging](logging.md) guide.
+
+## Environment Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DYN_LOGGING_JSONL` | Enable JSONL logging format (required for tracing) | `false` | `true` |
+| `OTEL_EXPORT_ENABLED` | Enable OTLP trace export | `false` | `true` |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | OTLP gRPC endpoint for Tempo | `http://localhost:4317` | `http://tempo:4317` |
+| `OTEL_SERVICE_NAME` | Service name for identifying components | `dynamo` | `dynamo-frontend` |
+
+## Getting Started Quickly
+
+### 1. Start Observability Stack
+
+Start the observability stack (Prometheus, Grafana, Tempo, exporters). See [Observability Getting Started](README.md#getting-started-quickly) for instructions.
+
+### 2. Set Environment Variables
+
+Configure Dynamo components to export traces:
+
+```bash
+# Enable JSONL logging and tracing
+export DYN_LOGGING_JSONL=true
+export OTEL_EXPORT_ENABLED=true
+export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317
+```
+
+### 3. Start Dynamo Components (Single GPU)
+
+For a simple single-GPU deployment, start the frontend and a single vLLM worker:
+
+```bash
+# Start the frontend with tracing enabled (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+export OTEL_SERVICE_NAME=dynamo-frontend
+python -m dynamo.frontend --router-mode kv &
+
+# Start a single vLLM worker (aggregated prefill and decode)
+export OTEL_SERVICE_NAME=dynamo-worker-vllm
+python -m dynamo.vllm --model Qwen/Qwen3-0.6B --enforce-eager \
+--otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT" &
+
+wait
+```
+
+This runs both prefill and decode on the same GPU, providing a simpler setup for testing tracing.
+
+### Alternative: Disaggregated Deployment (2 GPUs)
+
+Run the vLLM disaggregated script with tracing enabled:
+
+```bash
+# Navigate to vLLM launch directory
+cd examples/backends/vllm/launch
+
+# Export tracing env vars, then run the disaggregated deployment script.
+./disagg.sh
+```
+
+**Note:** the example vLLM `disagg.sh` sets additional per-worker port environment variables (e.g., `DYN_VLLM_KV_EVENT_PORT`,
+`VLLM_NIXL_SIDE_CHANNEL_PORT`) to avoid ZMQ "Address already in use" conflicts when multiple workers run on the same host. If you run the components manually, make sure you mirror those port settings.
+
+```bash
+#!/bin/bash
+set -e
+trap 'echo Cleaning up...; kill 0' EXIT
+
+# Enable tracing
+export DYN_LOGGING_JSONL=true
+export OTEL_EXPORT_ENABLED=true
+export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317
+
+# Run frontend (default port 8000, override with --http-port or DYN_HTTP_PORT env var)
+export OTEL_SERVICE_NAME=dynamo-frontend
+python -m dynamo.frontend --router-mode kv &
+
+# Run decode worker, make sure to wait for start up
+export OTEL_SERVICE_NAME=dynamo-worker-decode
+DYN_SYSTEM_PORT=8081 CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
+    --model Qwen/Qwen3-0.6B \
+    --enforce-eager \
+    --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT" &
+
+# Run prefill worker, make sure to wait for start up
+export OTEL_SERVICE_NAME=dynamo-worker-prefill
+DYN_SYSTEM_PORT=8082 \
+DYN_VLLM_KV_EVENT_PORT=20081 \
+VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \
+CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \
+    --model Qwen/Qwen3-0.6B \
+    --enforce-eager \
+    --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT" \
+    --is-prefill-worker &
+```
+
+For disaggregated deployments, this separates prefill and decode onto different GPUs for better resource utilization.
+
+### 4. Generate Traces
+
+Send requests to the frontend to generate traces (works for both aggregated and disaggregated deployments). **Note the `x-request-id` header**, which allows you to easily search for and correlate this specific trace in Grafana:
+
+```bash
+curl -H 'Content-Type: application/json' \
+-H 'x-request-id: test-trace-001' \
+-d '{
+  "model": "Qwen/Qwen3-0.6B",
+  "max_completion_tokens": 100,
+  "messages": [
+    {"role": "user", "content": "What is the capital of France?"}
+  ]
+}' \
+http://localhost:8000/v1/chat/completions
+```
+
+### 5. View Traces in Grafana Tempo
+
+1. Open Grafana at `http://localhost:3000`
+2. Login with username `dynamo` and password `dynamo`
+3. Navigate to **Explore** (compass icon in the left sidebar)
+4. Select **Tempo** as the data source (should be selected by default)
+5. In the query type, select **"Search"** (not TraceQL, not Service Graph)
+6. Use the **Search** tab to find traces:
+   - Search by **Service Name** (e.g., `dynamo-frontend`)
+   - Search by **Span Name** (e.g., `http-request`, `handle_payload`)
+   - Search by **Tags** (e.g., `x_request_id=test-trace-001`)
+7. Click on a trace to view the detailed flame graph
+
+#### Example Trace View
+
+Below is an example of what a trace looks like in Grafana Tempo:
+
+![Trace Example](../../assets/img/trace.png)
+
+### 6. Stop Services
+
+When done, stop the observability stack. See [Observability Getting Started](README.md#getting-started-quickly) for Docker Compose commands.
+
+---
+
+## Kubernetes Deployment
+
+For Kubernetes deployments, ensure you have a Tempo instance deployed and accessible (e.g., `http://tempo.observability.svc.cluster.local:4317`).
+
+### Modify DynamoGraphDeployment for Tracing
+
+Add common tracing environment variables at the top level and service-specific names in each component in your `DynamoGraphDeployment` (e.g., `examples/backends/vllm/deploy/disagg.yaml`):
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeployment
+metadata:
+  name: vllm-disagg
+spec:
+  # Common environment variables for all services
+  env:
+    - name: DYN_LOGGING_JSONL
+      value: "true"
+    - name: OTEL_EXPORT_ENABLED
+      value: "true"
+    - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
+      value: "http://tempo.observability.svc.cluster.local:4317"
+
+  services:
+    Frontend:
+      # ... existing configuration ...
+      extraPodSpec:
+        mainContainer:
+          # ... existing configuration ...
+          env:
+            - name: OTEL_SERVICE_NAME
+              value: "dynamo-frontend"
+
+    VllmDecodeWorker:
+      # ... existing configuration ...
+      extraPodSpec:
+        mainContainer:
+          # ... existing configuration ...
+          env:
+            - name: OTEL_SERVICE_NAME
+              value: "dynamo-worker-decode"
+
+    VllmPrefillWorker:
+      # ... existing configuration ...
+      extraPodSpec:
+        mainContainer:
+          # ... existing configuration ...
+          env:
+            - name: OTEL_SERVICE_NAME
+              value: "dynamo-worker-prefill"
+```
+
+Apply the updated DynamoGraphDeployment:
+
+```bash
+kubectl apply -f examples/backends/vllm/deploy/disagg.yaml
+```
+
+Traces will now be exported to Tempo and can be viewed in Grafana.
+

fern/pages/performance/aiconfigurator.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Finding Best Initial Configs using AIConfigurator"
+---
+
+[AIConfigurator](https://github.com/ai-dynamo/aiconfigurator/tree/main) is a performance optimization tool that helps you find the optimal configuration for deploying LLMs with Dynamo. It automatically determines the best number of prefill and decode workers, parallelism settings, and deployment parameters to meet your SLA targets while maximizing throughput.
+
+## Why Use AIConfigurator?
+
+When deploying LLMs with Dynamo, you need to make several critical decisions:
+- **Aggregated vs Disaggregated**: Which architecture gives better performance for your workload?
+- **Worker Configuration**: How many prefill and decode workers to deploy?
+- **Parallelism Settings**: What tensor/pipeline parallel configuration to use?
+- **SLA Compliance**: How to meet your TTFT and TPOT targets?
+
+AIConfigurator answers these questions in seconds, providing:
+- Optimal configurations that meet your SLA requirements
+- Ready-to-deploy Dynamo configuration files
+- Performance comparisons between different deployment strategies
+- Up to 1.7x better throughput compared to manual configuration
+
+## Quick Start
+
+```bash
+# Install
+pip3 install aiconfigurator
+
+# Find optimal configuration
+aiconfigurator cli default \
+  --model QWEN3_32B \        # Model name (QWEN3_32B, LLAMA3.1_70B, etc.)
+  --total_gpus 32 \          # Number of available GPUs
+  --system h200_sxm \        # GPU type (h100_sxm, h200_sxm, a100_sxm)
+  --isl 4000 \               # Input sequence length (tokens)
+  --osl 500 \                # Output sequence length (tokens)
+  --ttft 300 \               # Target Time To First Token (ms)
+  --tpot 10 \                # Target Time Per Output Token (ms)
+  --save_dir ./dynamo-configs
+
+# Deploy
+kubectl apply -f ./dynamo-configs/disagg/top1/disagg/k8s_deploy.yaml
+```
+
+## Example Output
+
+```text
+********************************************************************************
+*                     Dynamo aiconfigurator Final Results                      *
+********************************************************************************
+  ----------------------------------------------------------------------------
+  Input Configuration & SLA Target:
+    Model: QWEN3_32B (is_moe: False)
+    Total GPUs: 32
+    Best Experiment Chosen: disagg at 812.92 tokens/s/gpu (1.70x better)
+  ----------------------------------------------------------------------------
+  Overall Best Configuration:
+    - Best Throughput: 812.92 tokens/s/gpu
+    - User Throughput: 120.23 tokens/s/user
+    - TTFT: 276.76ms
+    - TPOT: 8.32ms
+  ----------------------------------------------------------------------------
+  Pareto Frontier:
+               QWEN3_32B Pareto Frontier: tokens/s/gpu vs tokens/s/user
+      ┌────────────────────────────────────────────────────────────────────────┐
+1600.0┤ •• disagg                                                              │
+      │ ff agg                                                                 │
+      │ xx disagg best                                                         │
+      │                                                                        │
+1333.3┤   f                                                                    │
+      │   ff                                                                   │
+      │     ff    •                                                            │
+      │       f   ••••••••                                                     │
+1066.7┤        f         ••                                                    │
+      │         fff       ••••••••                                             │
+      │            f              ••                                           │
+      │            f                ••••                                       │
+ 800.0┤            fffff                •••x                                   │
+      │                 fff                 ••                                 │
+      │                   fff                •                                 │
+      │                     fffff             ••                               │
+ 533.3┤                         ffff            ••                             │
+      │                             ffff          ••                           │
+      │                                 fffffff     •••••                      │
+      │                                        ffffff    ••                    │
+ 266.7┤                                              fffff •••••••••           │
+      │                                                   ffffffffff           │
+      │                                                             f          │
+      │                                                                        │
+   0.0┤                                                                        │
+      └┬─────────────────┬─────────────────┬────────────────┬─────────────────┬┘
+       0                60                120              180              240
+tokens/s/gpu                         tokens/s/user
+
+1. **Performance Comparison**: Shows disaggregated vs aggregated serving performance
+2. **Optimal Configuration**: The best configuration that meets your SLA targets
+3. **Deployment Files**: Ready-to-use Dynamo configuration files
+
+## Key Features
+
+### Fast Profiling Integration
+```bash
+# Use with Dynamo's SLA planner (20-30 seconds vs hours)
+python3 -m benchmarks.profiler.profile_sla \
+   --config ./examples/backends/trtllm/deploy/disagg.yaml \
+   --backend trtllm \
+   --use-ai-configurator \
+   --aic-system h200_sxm \
+   --aic-model-name QWEN3_32B
+```
+```
+
+### Custom Configuration
+```bash
+# For advanced users: define custom search space
+aiconfigurator cli exp --yaml_path custom_config.yaml
+```
+
+## Common Use Cases
+
+```bash
+# Strict SLAs (low latency)
+aiconfigurator cli default --model QWEN2.5_7B --total_gpus 8 --system h200_sxm --ttft 100 --tpot 5
+
+# High throughput (relaxed latency)
+aiconfigurator cli default --model QWEN3_32B --total_gpus 32 --system h200_sxm --ttft 1000 --tpot 50
+```
+
+## Supported Configurations
+
+**Models**: GPT, LLAMA2/3, QWEN2.5/3, Mixtral, DEEPSEEK_V3
+**GPUs**: H100, H200, A100, B200 (preview), GB200 (preview)
+**Backend**: TensorRT-LLM (vLLM and SGLang coming soon)
+
+## Additional Options
+
+```bash
+# Web interface
+aiconfigurator webapp  # Visit http://127.0.0.1:7860
+
+# Docker
+docker run -it --rm nvcr.io/nvidia/aiconfigurator:latest \
+  aiconfigurator cli default --model LLAMA3.1_70B --total_gpus 16 --system h100_sxm
+```
+
+## Troubleshooting
+
+**Model name mismatch**: Use exact model name that matches your deployment
+**GPU allocation**: Verify available GPUs match `--total_gpus`
+**Performance variance**: Results are estimates - benchmark actual deployment
+
+## Learn More
+
+- [Dynamo Installation Guide](../kubernetes/installation-guide.md)
+- [SLA Planner Quick Start Guide](../planner/sla-planner-quickstart.md)
+- [Benchmarking Guide](../benchmarks/benchmarking.md)
\ No newline at end of file

fern/pages/performance/tuning.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Disaggregation and Performance Tuning"
+---
+
+Disaggregation gains performance by separating the prefill and decode into different engines to reduce interferences between the two.
+However, performant disaggregation requires careful tuning of the inference parameters.
+Specifically, there are three sets of parameters that needs to be tuned:
+
+  1. Engine configuration and options (e.g. parallelization mapping, maximum number of tokens, etc.).
+  2. Disaggregated router configuration and options.
+  3. Number of prefill and decode engines.
+
+This guide describes the process of tuning these parameters.
+
+
+## Engine Configuration and Tuning
+
+The most important engine configuration to tune is the parallelization mapping.
+For most dense models, the best setting is to use TP within node and PP across nodes.
+For example, for Llama-405b w8a8 on H100, TP8 on a single node or TP8PP2 on two nodes is usually the best choice.
+The next thing to decide is how many numbers of GPU to serve the model.
+Typically, the number of GPUs vs the performance follows the following pattern:
+
+| Number of GPUs                                      | Performance
+| :-------------------------------------------------- | :---------------------------------------------------------------------------------------- |
+| Cannot hold weights in VRAM                         | OOM                                                                                       |
+| (Barely hold weights in VRAM)                       | (KV cache is too small to maintain large enough sequence length or reasonable batch size) |
+| Minimum number with fair amount of KV cache         | Best overall throughput/GPU, worst latency/user                                           |
+| Between minimum and maximum                         | Tradeoff between throughput/GPU and latency/user                                          |
+| Maximum number limited by communication scalability | Worst overall throughput/GPU, best latency/user                                           |
+| More than maximum                                   | Communication overhead dominates, poor performance                                        |
+
+<Note>
+for decode-only engines, sometimes larger number of GPUs has to larger KV cache per GPU and more decoding requests running in parallel, which leads to both better throughput/GPU and better latency/user.
+For example, for Llama-3.3-70b NVFP4 quantization on B200 in vLLM with 0.9 free GPU memory fraction:
+</Note>
+
+| TP Size | KV Cache Size (GB) | KV Cache per GPU (GB) | Per GPU Improvement over TP1 |
+| ------: | -----------------: | --------------------: | ---------------------------: |
+|       1 |                113 |                   113 |                        1.00x |
+|       2 |                269 |                   135 |                        1.19x |
+|       4 |                578 |                   144 |                        1.28x |
+
+The best number of GPUs to use in the prefill and decode engines can be determined by running a few fixed ISL/OSL/concurrency test using [AIPerf](https://github.com/ai-dynamo/aiperf/tree/main) and compare with the SLA.
+AIPerf is pre-installed in the dynamo container.
+
+<Tip>
+If you are unfamiliar with AIPerf, please see this helpful [tutorial](https://github.com/ai-dynamo/aiperf/blob/main/docs/tutorial.md) to get you started.
+</Tip>
+
+Besides the parallelization mapping, other common knobs to tune are maximum batch size, maximum number of tokens, and block size.
+For prefill engines, usually a small batch size and large `max_num_token` is preferred.
+For decode engines, usually a large batch size and medium `max_num_token` is preferred.
+For details on tuning the `max_num_token` and max_batch_size, see the next section.
+
+For block size, if the block size is too small, it leads to small memory chunks in the P->D KV cache transfer and poor performance.
+Too small block size also leads to memory fragmentation in the attention calculation, but the impact is usually insignificant.
+If the block size is too large, it leads to low prefix cache hit ratio.
+For most dense models, we find block size 128 is a good choice.
+
+
+## Disaggregated Router
+
+Disaggregated router decides whether to prefill a request in the remote prefill engine or locally in the decode engine using chunked prefill.
+For most frameworks, when chunked prefill is enabled and one forward iteration gets a mixture of prefilling and decoding request, three kernels are launched:
+
+  1. The attention kernel for context tokens (context_fmha kernel in TRTLLM).
+
+  2. The attention kernel for decode tokens (xqa kernel in TRTLLM).
+
+  3. Dense kernel for the combined active tokens in prefills and decodes.
+
+### Prefill Engine
+
+In the prefill engine, the best strategy is to operate at the smallest batch size that saturates the GPUs so that the average time to first token (TTFT) is minimized.
+For example, for Llama3.3-70b NVFP4 quantization on B200 TP1 in vLLM, the below figure shows the prefill time with different isl (prefix caching is turned off):
+
+![Combined bar and line chart showing "Prefill Time". Bar chart represents TTFT (Time To First Token) in milliseconds against ISL (Input Sequence Length). The line chart shows TTFT/ISL (milliseconds per token) against ISL.](../../assets/img/prefill-time.png)
+
+For isl less than 1000, the prefill efficiency is low because the GPU is not fully saturated.
+For isl larger than 4000, the prefill time per token increases because the attention takes longer to compute with a longer history.
+
+Currently, prefill engines in Dynamo operate at a batch size of 1.
+To make sure prefill engine is saturated, users can set `max-local-prefill-length` to the saturation point to make sure prefill engine is optimal.
+
+### Decode Engine
+
+In the decode engine, maximum batch size and maximum number of tokens affects the size of intermediate tensors.
+With a larger batch size and number of tokens, the size of intermediate tensors increases and the size of KV cache decreases.
+TensorRT-LLM (TRTLLM) has a good [summary](https://nvidia.github.io/TensorRT-LLM/reference/memory.html) on the memory footprint where similar ideas also applies to other LLM frameworks.
+
+With chunked prefill enabled, the maximum number of tokens controls the longest prefill that can be piggybacked to decode and control the inter-token latency (ITL).
+For the same prefill requests, a large maximum number of tokens leads to fewer but longer stalls in the generation, while a small maximum number of tokens leads to more but shorter stalls in the generation.
+However, chunked prefill is currently not supported in Dynamo (vLLM backend).
+Hence, the current best strategy is to set the maximum batch size to the optimized KV cache size and set the maximum number of tokens to the maximum local prefill length + maximum batch size (since one decode request has one active token).
+
+
+## Number of Prefill and Decode Engines
+
+The best dynamo knob choices depends on the operating condition of the model.
+Based on the load, we define three operating conditions:
+
+  1. **Low load**:
+     The endpoint is hit by a single user (single-stream) most of the time.
+
+  2. **Medium load**:
+     The endpoint is hit by multiple users, but the KV cache of the decode engines is never fully utilized.
+
+  3. **High load**:
+     The endpoint is hit by multiple users and the requests are queued up due to no available KV cache in the decode engines.
+
+At low load, disaggregation would not benefit much as prefill and decode are usually computed separately.
+It is usually better to use a single monolithic engine.
+
+At medium load, disaggregation allows better ITL compared with prefill-prioritized and chunked prefill engines and better TTFT compared with chunked prefill engine and decode-only engine for each user.
+Dynamo users can adjust the number of prefill and decode engines based on TTFT and ITL SLA.
+
+At high load where KV cache capacity is the bottleneck, disaggregation has the following effect on the KV cache usage in the decode engines:
+
+  * Increase the total amount of KVcache:
+
+    * Being able to use greater TP values in decode engines leads to more KV cache per GPU and higher prefix cache hit rate.
+
+    * When the requests is prefilled remotely, the decode engine does not need to maintain its KV cache (currently not implemented in Dynamo).
+
+    * Lower ITL reduces the decode time and allow the same amount of KV cache to serve more requests.
+
+  * Decrease the total amount of KV cache:
+
+    * Some GPUs are configured as prefill engines whose KV cache is not used in the decode phase.
+
+Since Dynamo currently allocates the KV blocks immediately when the decode engine get the requests,
+it is advisable to use as few prefill engines as possible (even no prefill engine) to maximize the available KV cache in decode engines.
+To prevent queueing at prefill engines, users can set a large `max-local-prefill-length` and piggyback more prefill requests at decode engines.

fern/pages/planner/load-planner.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Load-based Planner"
+---
+
+This document covers load-based planner in `examples/llm/components/planner.py`.
+
+<Warning>
+Load-based planner is inoperable as vllm, sglang, and trtllm examples all do not use prefill queues. Please use SLA planner for now.
+</Warning>
+
+<Warning>
+Bare metal deployment with local connector is deprecated. The only option to deploy load-based planner is via k8s. We will update the examples in this document soon.
+</Warning>
+
+## Load-based Scaling Up/Down Prefill/Decode Workers
+
+To adjust the number of prefill/decode workers, planner monitors the following metrics:
+* Prefill worker: planner monitors the number of requests pending in the prefill queue to estimate the prefill workload.
+* Decode/aggregated worker: planner monitors the average KV cache utilization rate to estimate the decode/aggregated workload.
+
+Every `metric-pulling-interval`, planner gathers the aforementioned metrics. Every `adjustment-interval`, planner compares the aggregated metrics in this interval with pre-set thresholds and decide to scale up/down prefill/decode workers. To avoid over-compensation, planner only changes the number of workers by 1 in one adjustment interval. In addition, when the number of workers is being adjusted, the planner blocks the metric pulling and adjustment.
+
+To scale up a prefill/decode worker, planner just need to launch the worker in the correct namespace. The auto-discovery mechanism picks up the workers and add them to the routers. To scale down a prefill worker, planner send a SIGTERM signal to the prefill worker. The prefill worker store the signal and exit when it finishes the current request pulled from the prefill queue. This ensures that no remote prefill request is dropped. To scale down a decode worker, planner revokes the etcd lease of the decode worker. When the etcd lease is revoked, the corresponding decode worker is immediately removed from the router and won't get any new requests. The decode worker then finishes all the current requests in their original stream and exits gracefully.
+
+There are two additional rules set by planner to prevent over-compensation:
+1. After a new decode worker is added, since it needs time to populate the kv cache, planner doesn't scale down the number of decode workers in the next `NEW_DECODE_WORKER_GRACE_PERIOD=3` adjustment intervals.
+1. We do not scale up prefill worker if the prefill queue size is estimated to reduce below the `--prefill-queue-scale-up-threshold` within the next `NEW_PREFILL_WORKER_QUEUE_BUFFER_PERIOD=3` adjustment intervals following the trend observed in the current adjustment interval.
+
+## SLA-based Scaling Up/Down Prefill/Decode Workers
+
+See [SLA-Driven Profiling](../benchmarks/sla-driven-profiling.md) for more details.
+
+## Usage
+
+The planner integration with the new frontend + worker architecture is currently a work in progress. This documentation will be updated with the new deployment patterns and code examples once the planner component has been fully adapted to the new workflow.
+
+Configuration options:
+* `namespace` (str, default: "dynamo"): Target namespace for planner operations
+* `environment` (str, default: "local"): Target environment (local, kubernetes)
+* `no-operation` (bool, default: false): Run in observation mode only
+* `log-dir` (str, default: None): Tensorboard log directory
+* `adjustment-interval` (int, default: 30): Seconds between adjustments
+* `metric-pulling-interval` (int, default: 1): Seconds between metric pulls
+* `max-gpu-budget` (int, default: 8): Maximum GPUs for all workers
+* `min-gpu-budget` (int, default: 1): Minimum GPUs per worker type
+* `decode-kv-scale-up-threshold` (float, default: 0.9): KV cache threshold for scale-up
+* `decode-kv-scale-down-threshold` (float, default: 0.5): KV cache threshold for scale-down
+* `prefill-queue-scale-up-threshold` (float, default: 0.5): Queue threshold for scale-up
+* `prefill-queue-scale-down-threshold` (float, default: 0.2): Queue threshold for scale-down
+* `decode-engine-num-gpu` (int, default: 1): GPUs per decode engine
+* `prefill-engine-num-gpu` (int, default: 1): GPUs per prefill engine
+
+Run as standalone process:
+```bash
+PYTHONPATH=/workspace/examples/llm python components/planner.py --namespace=dynamo --served-model-name=vllm --no-operation --log-dir=log/planner
+```
+
+Monitor metrics with Tensorboard:
+```bash
+tensorboard --logdir=<path-to-tensorboard-log-dir>
+```

fern/pages/planner/planner-intro.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "Planner"
+---
+
+The planner monitors the state of the system and adjusts workers to
+ensure that the system runs efficiently.
+
+Currently, the planner can scale the number of vllm workers up and down
+based on the kv cache load and prefill queue size:
+
+Key features include:
+
+- **SLA-based scaling** that uses predictive modeling and performance
+  interpolation to proactively meet TTFT and ITL targets
+- **Graceful scaling** that ensures no requests are dropped during
+  scale-down operations
+
+<Tip>
+**New to SLA Planner?** Start with the [SLA Planner Quick Start Guide](sla-planner-quickstart.md) for a complete, step-by-step workflow.
+
+**Prerequisites**: SLA-based planner requires pre-deployment profiling (2-4 hours on real silicon or a few minutes using simulator) before deployment. The Quick Start guide includes everything you need.
+</Tip>

fern/pages/planner/sla-planner-quickstart.md

+---
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+title: "SLA-Driven Profiling and Planner Deployment Quick Start Guide"
+---
+
+Complete workflow to deploy SLA-optimized Dynamo models using DynamoGraphDeploymentRequests (DGDR). This guide shows how to automatically profile models and deploy them with optimal configurations that meet your Service Level Agreements (SLAs).
+
+<Warning>
+**Prerequisites**: This guide assumes you have a Kubernetes cluster with GPU nodes and have completed the [Dynamo Platform installation](../kubernetes/installation-guide.md).
+</Warning>
+
+## Overview
+
+The DGDR workflow automates the entire process from SLA specification to deployment:
+
+1. **Define SLAs**: Specify performance requirements (TTFT, ITL) and model information in a DGDR Custom Resource
+2. **Automatic Profiling**: The Dynamo Operator automatically profiles your model to find optimal configurations
+3. **Auto-Deploy**: The system automatically deploys the optimal configuration that meets your SLAs
+
+```mermaid
+flowchart TD
+    A[Create DGDR] --> B[DGDR Controller]
+    B --> C{Profiling Method}
+    C -->|Online| D[Run Profiling Job<br/>2-4 hours]
+    C -->|Offline/AIC| E[AI Configurator<br/>20-30 seconds]
+    D --> F[Generate DGD Config]
+    E --> F
+    F --> G[Auto-Deploy DGD]
+    G --> H[Monitor & Scale]
+
+    style A fill:#e1f5fe
+    style D fill:#fff3e0
+    style E fill:#e8f5e8
+    style G fill:#f3e5f5
+    style H fill:#fff8e1
+```
+
+## What is a DynamoGraphDeploymentRequest (DGDR)?
+
+A **DynamoGraphDeploymentRequest (DGDR)** is a Kubernetes Custom Resource that serves as the primary interface for users to request model deployments with specific performance and resource constraints. Think of it as a "deployment order" where you specify:
+
+- **What** model you want to deploy (`model`)
+- **How** it should perform (SLA targets: `ttft`, `itl`)
+- **Where** it should run (optional GPU preferences)
+- **Which** backend to use (`backend`: vllm, sglang, or trtllm)
+- **Which** images to use (`profilingConfig.profilerImage`, `deploymentOverrides.workersImage`)
+
+The Dynamo Operator watches for DGDRs and automatically:
+1. Discovers available GPU resources in your cluster
+2. Runs profiling (online or offline) to find optimal configurations
+3. Generates an optimized DynamoGraphDeployment (DGD) configuration
+4. Deploys the DGD to your cluster
+
+**Key Benefits:**
+- **Declarative**: Specify what you want, not how to achieve it
+- **Automated**: No manual profiling job setup or result processing
+- **SLA-Driven**: Ensures deployments meet your performance requirements
+- **Integrated**: Works seamlessly with the Dynamo Operator
+
+## Prerequisites
+
+Before creating a DGDR, ensure:
+- **Dynamo platform installed** with the operator running (see [Installation Guide](../kubernetes/installation-guide.md))
+- **[kube-prometheus-stack](../kubernetes/observability/metrics.md) installed and running** (required for SLA planner)
+- **Image pull secrets configured** if using private registries (typically `nvcr-imagepullsecret` for NVIDIA images)
+- **Sufficient GPU resources** available in your cluster for profiling
+- **Runtime images available** that contain both profiler and runtime components
+
+### Container Images
+
+Each DGDR requires you to specify container images for the profiling and deployment process:
+
+**profilingConfig.profilerImage** (Required):
+Specifies the container image used for the profiling job itself. This image must contain the profiler code and dependencies needed for SLA-based profiling.
+
+**deploymentOverrides.workersImage** (Optional):
+Specifies the container image used for DynamoGraphDeployment worker components (frontend, workers, planner). This image is used for:
+- Temporary DGDs created during online profiling (for performance measurements)
+- The final DGD deployed after profiling completes
+
+If `workersImage` is omitted, the image from the base config file (e.g., `disagg.yaml`) is used. You may use our public images (0.6.1 and later) or build and push your own.
+
+```yaml
+spec:
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"  # Optional
+```
+
+## Quick Start: Deploy with DGDR
+
+### Step 1: Create Your DGDR
+
+Dynamo provides sample DGDR configurations in `benchmarks/profiler/deploy/`. You can use these as starting points:
+
+**Available Sample DGDRs:**
+- **`profile_sla_dgdr.yaml`**: Standard online profiling for dense models
+- **`profile_sla_aic_dgdr.yaml`**: Fast offline profiling using AI Configurator (TensorRT-LLM)
+- **`profile_sla_moe_dgdr.yaml`**: Online profiling for MoE models (SGLang)
+
+Or, you can create your own DGDR for your own needs:
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: my-model-deployment  # Change the name
+  namespace: default         # Change the namespace
+spec:
+  model: "Qwen/Qwen3-0.6B"     # Update to your model
+  backend: vllm                # Backend: vllm, sglang, or trtllm
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"  # Required
+    config:
+      sla:
+        isl: 3000    # Adjust to your workload
+        osl: 150     # Adjust to your workload
+        ttft: 200    # Your target (ms)
+        itl: 20      # Your target (ms)
+
+      sweep:
+        use_ai_configurator: false  # Set to true for fast profiling (TensorRT-LLM only)
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"  # Optional
+
+  autoApply: true  # Auto-deploy after profiling
+```
+
+<Tip>
+For detailed explanations of all configuration options (SLA, hardware, sweep, AIC, planner), see the [DGDR Configuration Reference](../benchmarks/sla-driven-profiling.md#dgdr-configuration-reference).
+</Tip>
+
+### Step 2: Apply the DGDR
+
+The rest of this quickstart will use the DGDR sample that uses AIC profiling. If you use a different DGDR file and/or name, be sure to adjust the commands accordingly.
+
+```bash
+export NAMESPACE=your-namespace
+kubectl apply -f benchmarks/profiler/deploy/profile_sla_aic_dgdr.yaml -n $NAMESPACE
+```
+
+The Dynamo Operator will immediately begin processing your request.
+
+### Step 3: Monitor Progress
+
+Watch the DGDR status:
+
+```bash
+# View status
+kubectl get dgdr -n $NAMESPACE
+
+# Detailed status
+kubectl describe dgdr sla-aic -n $NAMESPACE
+
+# Watch profiling job logs
+kubectl logs -f job/profile-sla-aic -n $NAMESPACE
+```
+
+**DGDR Status States:**
+- `Pending`: Initial state, preparing to profile
+- `Profiling`: Running profiling job (20-30 seconds for AIC, 2-4 hours for online)
+- `Deploying`: Generating and applying DGD configuration
+- `Ready`: DGD successfully deployed and running
+- `Failed`: Error occurred (check events for details)
+
+<Note>
+With AI Configurator, profiling completes in **20-30 seconds**! This is much faster than online profiling which takes 2-4 hours.
+</Note>
+
+### Step 4: Access Your Deployment
+
+Once the DGDR reaches `Ready` state, your model is deployed and ready to serve:
+
+```bash
+# Find the frontend service
+kubectl get svc -n $NAMESPACE | grep trtllm-disagg
+
+# Port-forward to access locally
+kubectl port-forward svc/trtllm-disagg-frontend 8000:8000 -n $NAMESPACE
+
+# Test the endpoint
+curl http://localhost:8000/v1/models
+```
+
+### Step 5 (Optional): Access the Planner Grafana Dashboard
+
+If you want to monitor the SLA Planner's decision-making in real-time, you can deploy the Planner Grafana dashboard.
+
+```bash
+kubectl apply -n monitoring -f deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml
+```
+
+Follow the instructions in [Dynamo Metrics Collection on Kubernetes](../kubernetes/observability/metrics.md) to access the Grafana UI and select the **Dynamo Planner Dashboard**.
+
+The dashboard displays:
+- **Worker Counts & GPU Usage**: Current prefill/decode worker counts and cumulative GPU hours
+- **Observed Metrics**: Real-time TTFT, ITL, request rate, and sequence lengths from Prometheus
+- **Predicted Metrics**: Planner's load predictions and recommended replica counts
+- **Correction Factors**: How the planner adjusts predictions based on observed vs expected performance
+
+<Tip>
+Use the **Namespace** dropdown at the top of the dashboard to filter metrics for your specific deployment namespace.
+</Tip>
+
+## DGDR Configuration Details
+
+### Required Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `spec.model` | string | Model identifier (e.g., "meta-llama/Llama-3-70b") |
+| `spec.backend` | enum | Inference backend: `vllm`, `sglang`, or `trtllm` |
+| `spec.profilingConfig.profilerImage` | string | Container image for profiling job |
+| `spec.profilingConfig.config.sla` | object | SLA targets (isl, osl, ttft, itl) |
+
+### Optional Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `spec.deploymentOverrides.workersImage` | string | Container image for DGD worker components. If omitted, uses image from base config file. |
+| `spec.autoApply` | boolean | Automatically deploy DGD after profiling (default: false) |
+| `spec.deploymentOverrides` | object | Customize metadata (name, namespace, labels, annotations) and image for auto-created DGD |
+
+### SLA Configuration
+
+The `sla` section defines performance requirements and workload characteristics:
+
+```yaml
+sla:
+  isl: 3000      # Average input sequence length (tokens)
+  osl: 150       # Average output sequence length (tokens)
+  ttft: 200      # Target Time To First Token (milliseconds, float)
+  itl: 20        # Target Inter-Token Latency (milliseconds, float)
+```
+
+**Choosing SLA Values:**
+- **ISL/OSL**: Based on your expected traffic patterns
+- **TTFT**: First token latency target (lower = more GPUs needed)
+- **ITL**: Token generation latency target (lower = more GPUs needed)
+- **Trade-offs**: Tighter SLAs require more GPU resources
+
+### Profiling Methods
+
+Choose between **online profiling** (real measurements, 2-4 hours) or **offline profiling** with AI Configurator (estimated, 20-30 seconds):
+
+```yaml
+# Online Profiling (Default)
+sweep:
+  use_ai_configurator: false
+
+# Offline Profiling (AI Configurator - TensorRT-LLM only)
+sweep:
+  use_ai_configurator: true
+  aic_system: h200_sxm
+  aic_hf_id: Qwen/Qwen3-32B
+  aic_backend_version: "0.20.0"
+```
+
+<Note>
+For detailed comparison, supported configurations, and limitations, see [SLA-Driven Profiling Documentation](../benchmarks/sla-driven-profiling.md#profiling-method).
+</Note>
+
+### Hardware Configuration
+
+For details on hardware configuration and GPU discovery options, see [Hardware Configuration in SLA-Driven Profiling](../benchmarks/sla-driven-profiling.md#hardware-configuration).
+
+### Advanced Configuration
+
+#### Using Existing DGD Configs (Recommended for Custom Setups)
+
+If you have an existing DynamoGraphDeployment config (e.g., from `examples/backends/*/deploy/disagg.yaml` or custom recipes), you can reference it via ConfigMap:
+
+**Step 1: Create ConfigMap from your DGD config file:**
+
+```bash
+kubectl create configmap deepseek-r1-config \
+  --from-file=disagg.yaml=/path/to/your/disagg.yaml \
+  --namespace $NAMESPACE \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+
+**Step 2: Reference the ConfigMap in your DGDR:**
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: deepseek-r1
+spec:
+  model: deepseek-ai/DeepSeek-R1
+  backend: sglang
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
+    configMapRef:
+      name: deepseek-r1-config
+      key: disagg.yaml  # Must match the key used in --from-file
+    config:
+      sla:
+        isl: 4000
+        osl: 500
+        ttft: 300
+        itl: 10
+      sweep:
+        use_ai_configurator: true
+      aic:
+        system: h200_sxm
+        model_name: DEEPSEEK_V3
+        backend_version: "0.20.0"
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
+
+  autoApply: true
+```
+
+> **What's happening**: The profiler uses the DGD config from the ConfigMap as a **base template**, then optimizes it based on your SLA targets. The controller automatically injects `spec.model` into `deployment.model` and `spec.backend` into `engine.backend` in the final configuration.
+
+#### Inline Configuration (Simple Use Cases)
+
+For simple use cases without a custom DGD config, provide profiler configuration directly. The profiler will auto-generate a basic DGD configuration from your `model` and `backend`:
+
+```yaml
+profilingConfig:
+  config:
+    # SLA targets (required for profiling)
+    sla:
+      isl: 8000   # Input sequence length
+      osl: 200    # Output sequence length
+      ttft: 200.0 # Time To First Token (ms)
+      itl: 10.0   # Inter-Token Latency (ms)
+
+    # Hardware constraints (optional)
+    hardware:
+      min_num_gpus_per_engine: 2
+      max_num_gpus_per_engine: 8
+      gpu_type: h200_sxm
+
+    # Profiling sweep settings (optional)
+    sweep:
+      prefill_interpolation_granularity: 16  # Number of samples for prefill ISL sweep
+      decode_interpolation_granularity: 6    # Number of samples for decode sweep
+```
+
+> **Note**: `engine.config` is a **file path** to a DGD YAML file, not inline configuration. Use ConfigMapRef (recommended) or leave it unset to auto-generate.
+
+#### Planner Configuration Passthrough
+Add planner-specific settings. Planner arguments use a `planner_` prefix:
+
+```yaml
+profilingConfig:
+  config:
+    planner:
+      planner_min_endpoint: 2
+```
+
+## Understanding Profiling Results
+
+For details about the profiling process, performance plots, and interpolation data, see [SLA-Driven Profiling Documentation](../benchmarks/sla-driven-profiling.md).
+
+## Advanced Topics
+
+### Mocker Deployment
+
+Instead of a real DGD that uses GPU resources, you can deploy a mocker deployment that uses simulated engines rather than GPUs. Mocker is available in all backend images and uses profiling data to simulate realistic GPU timing behavior. It is useful for:
+- Large-scale experiments without GPU resources
+- Testing Planner behavior and infrastructure
+- Validating deployment configurations
+
+To deploy mocker instead of the real backend, set `useMocker: true`:
+
+```yaml
+spec:
+  model: <model-name>
+  backend: trtllm  # Real backend for profiling (vllm, sglang, or trtllm)
+  useMocker: true  # Deploy mocker instead of real backend
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/dynamo/trtllm-runtime:<image-tag>"
+    ...
+  autoApply: true
+```
+
+Profiling still runs against the real backend (via GPUs or AIC) to collect performance data. The mocker deployment then uses this data to simulate realistic timing behavior.
+
+### DGDR Immutability
+
+DGDRs are **immutable** - if you need to update SLAs or configuration:
+
+1. Delete the existing DGDR: `kubectl delete dgdr sla-aic`
+2. Create a new DGDR with updated specifications
+
+### Manual Deployment Control
+
+There are two ways to manually control deployment after profiling:
+
+#### Option 1: Use DGDR-Generated Configuration (Recommended)
+
+Disable auto-deployment to review the generated DGD before applying:
+
+```yaml
+spec:
+  autoApply: false
+```
+
+Then manually extract and apply the generated DGD:
+
+```bash
+# Extract generated DGD from DGDR status
+kubectl get dgdr sla-aic -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' | kubectl apply -f -
+
+# Or save to file first for review/modification
+kubectl get dgdr sla-aic -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' > my-dgd.yaml
+
+vi my-dgd.yaml
+kubectl apply -f my-dgd.yaml -n $NAMESPACE
+```
+
+The generated DGD includes optimized configurations and the SLA planner component. The required `planner-profile-data` ConfigMap is automatically created when profiling completes, so the DGD will deploy successfully.
+
+#### Option 2: Use Standalone Planner Templates (Advanced)
+
+For advanced use cases, you can manually deploy using the standalone planner templates in `examples/backends/*/deploy/disagg_planner.yaml`:
+
+```bash
+# After profiling completes, profiling data is automatically stored in ConfigMaps
+
+# OPTIONAL: Inspect profiling results stored in ConfigMaps
+# View the generated DGD configuration
+kubectl get configmap dgdr-output-<dgdr-name> -n $NAMESPACE -o yaml
+
+# View the planner profiling data (JSON format)
+kubectl get configmap planner-profile-data -n $NAMESPACE -o yaml
+
+# Update the PROMETHEUS_ENDPOINT environment variable in the planner template
+# to match your cluster's Prometheus service location (see comments in the template)
+
+# Update backend planner manifest as needed, then deploy
+kubectl apply -f examples/backends/<backend>/deploy/disagg_planner.yaml -n $NAMESPACE
+```
+
+> **Note**: The standalone templates are provided as examples and may need customization for your model and requirements. The DGDR-generated configuration (Option 1) is recommended as it's automatically tuned to your profiling results and SLA targets.
+>
+> **Important - Prometheus Configuration**: The planner queries Prometheus to get frontend request metrics for scaling decisions. If you see errors like "Failed to resolve prometheus service", ensure the `PROMETHEUS_ENDPOINT` environment variable in your planner configuration correctly points to your Prometheus service. See the comments in the example templates for details.
+
+### Relationship to DynamoGraphDeployment (DGD)
+
+- **DGDR**: High-level "intent" - what you want deployed
+- **DGD**: Low-level "implementation" - how it's deployed
+
+The DGDR controller generates a DGD that:
+- Uses optimal TP configurations from profiling
+- Includes SLA planner for autoscaling
+- Has deployment and engine settings tuned for your SLAs
+
+The generated DGD is tracked via labels:
+```yaml
+metadata:
+  labels:
+    dgdr.nvidia.com/name: sla-aic
+    dgdr.nvidia.com/namespace: your-namespace
+```
+
+### Accessing Detailed Profiling Artifacts
+
+By default, profiling jobs save essential data to ConfigMaps for planner integration. For advanced users who need access to detailed artifacts (logs, performance plots, AIPerf results, etc), configure the DGDR to use `dynamo-pvc`. This is optional and will not affect the functionality of profiler or Planner.
+
+**What's available in ConfigMaps (always created):**
+- Generated DGD configuration
+- Profiling data for Planner (`.json` files)
+
+**What's available in PVC if attached to DGDR (optional):**
+- Performance plots (PNGs)
+- DGD configuration and logs of all services for each profiled deployment
+- AIPerf profiling artifacts for each AIPerf run
+- Raw profiling data (`.npz` files)
+- Profiler log
+
+**Setup:**
+
+1. Set up the benchmarking PVC:
+```bash
+export NAMESPACE=your-namespace
+deploy/utils/setup_benchmarking_resources.sh
+```
+
+2. Add `outputPVC` to your DGDR's `profilingConfig`:
+```yaml
+spec:
+  profilingConfig:
+    outputPVC: "dynamo-pvc"
+    config:
+      # ... rest of config
+```
+
+3. After profiling completes, access results:
+```bash
+kubectl apply -f deploy/utils/manifests/pvc-access-pod.yaml -n $NAMESPACE
+kubectl wait --for=condition=Ready pod/pvc-access-pod -n $NAMESPACE --timeout=60s
+kubectl cp $NAMESPACE/pvc-access-pod:/data ./profiling-results
+kubectl delete pod pvc-access-pod -n $NAMESPACE
+```
+
+## Troubleshooting
+
+### Quick Diagnostics
+
+```bash
+# Check DGDR status and events
+kubectl describe dgdr sla-aic -n $NAMESPACE
+
+# Check operator logs
+kubectl logs -n $NAMESPACE -l app.kubernetes.io/name=dynamo-operator --tail=100
+
+# Check profiling job logs
+kubectl logs -l job-name=profile-sla-aic -n $NAMESPACE
+```
+
+### Common Issues
+
+| Issue | Quick Fix |
+|-------|-----------|
+| **DGDR stuck in Pending** | Check GPU availability: `kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}'` |
+| **Image pull errors** | Verify secret exists: `kubectl get secret nvcr-imagepullsecret -n $NAMESPACE` |
+| **Profiling fails** | Check job logs: `kubectl logs -l job-name=profile-sla-aic -n $NAMESPACE` |
+| **SLA cannot be met** | Relax TTFT/ITL targets or add more GPUs |
+| **DGD not deployed** | Verify `autoApply: true` in DGDR spec |
+
+<Note>
+For comprehensive troubleshooting including AI Configurator constraints, performance debugging, and backend-specific issues, see [SLA-Driven Profiling Troubleshooting](../benchmarks/sla-driven-profiling.md#troubleshooting).
+</Note>
+
+## Configuration Reference
+
+For comprehensive documentation of all DGDR configuration options, see the [DGDR Configuration Reference](../benchmarks/sla-driven-profiling.md#dgdr-configuration-reference).
+
+This includes detailed explanations of:
+- **SLA Configuration**: ISL, OSL, TTFT, ITL with use cases and trade-offs
+- **Hardware Configuration**: GPU constraints and search space control
+- **Sweep Configuration**: Profiling behavior and interpolation settings
+- **AI Configurator Configuration**: System types, model mappings, backend versions
+- **Planner Configuration**: Autoscaling and adjustment parameters
+- **Complete Examples**: Full DGDRs for online, offline (AIC), and MoE profiling
+
+## Related Documentation
+
+- [DGDR API Reference](../kubernetes/api-reference.md)
+- [Pre-Deployment Profiling Details](../benchmarks/sla-driven-profiling.md)
+- [SLA Planner Architecture](sla-planner.md)
+- [Dynamo Operator Guide](../kubernetes/dynamo-operator.md)