docs: migrate Profiler docs to three-tier structure (#6003)

Signed-off-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: dagil-nvidia <dagil@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jonathan Tong <jt572@cornell.edu>

docs/backends/sglang/profiling.md

@@ -5,6 +5,9 @@ SPDX-License-Identifier: Apache-2.0
 
 # Profiling SGLang Workers in Dynamo
 
+> [!NOTE]
+> **See also**: [Profiler Component Overview](/docs/components/profiler/README.md) for SLA-driven profiling and deployment optimization.
+
 Dynamo exposes profiling endpoints for SGLang workers via the system server's `/engine/*` routes. This allows you to start and stop PyTorch profiling on running inference workers without restarting them.
 
 These endpoints wrap SGLang's internal `TokenizerManager.start_profile()` and `stop_profile()` methods. See SGLang's documentation for the full list of supported parameters.

docs/benchmarks/sla_driven_profiling.md

@@ -3,6 +3,9 @@
 > [!TIP]
 > **New to DGDR and SLA-Driven Profiling?** Start with the [SLA-Driven Profiling and Planner Deployment Quick Start Guide](/docs/planner/sla_planner_quickstart.md) for step-by-step instructions. This document provides deeper technical details about the profiling process.
 
+> [!NOTE]
+> **See also**: [Profiler Component Overview](/docs/components/profiler/README.md) for a quick start guide and feature matrix.
+
 ## Overview
 
 Dynamo provides automated SLA-driven profiling through **DynamoGraphDeploymentRequests (DGDR)**. Instead of manually running profiling scripts, you declare your performance requirements and let the Dynamo Operator handle profiling and deployment automatically.

docs/components/profiler/README.md

+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Profiler
+
+The Dynamo Profiler is an automated performance analysis tool that measures model inference characteristics to optimize deployment configurations. It determines optimal tensor parallelism (TP) settings for prefill and decode phases, generates performance interpolation data, and enables SLA-driven autoscaling through the Planner.
+
+## Feature Matrix
+
+| Feature | vLLM | SGLang | TensorRT-LLM |
+|---------|------|--------|--------------|
+| Dense Model Profiling | ✅ | ✅ | ✅ |
+| MoE Model Profiling | 🚧 | ✅ | 🚧 |
+| AI Configurator (Offline) | ❌ | ❌ | ✅ |
+| Online Profiling (AIPerf) | ✅ | ✅ | ✅ |
+| Interactive WebUI | ✅ | ✅ | ✅ |
+| Runtime Profiling Endpoints | ❌ | ✅ | ❌ |
+
+## Quick Start
+
+### Prerequisites
+
+- Dynamo platform installed (see [Installation Guide](/docs/kubernetes/installation_guide.md))
+- Kubernetes cluster with GPU nodes (for DGDR-based profiling)
+- kube-prometheus-stack installed (required for SLA planner)
+
+### Using DynamoGraphDeploymentRequest (Recommended)
+
+The recommended way to profile models is through DGDRs, which automate the entire profiling and deployment workflow.
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: my-model-profiling
+spec:
+  model: "Qwen/Qwen3-0.6B"
+  backend: vllm
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+    config:
+      sla:
+        isl: 3000      # Average input sequence length
+        osl: 150       # Average output sequence length
+        ttft: 200.0    # Target Time To First Token (ms)
+        itl: 20.0      # Target Inter-Token Latency (ms)
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+
+  autoApply: true
+```
+
+```bash
+kubectl apply -f my-profiling-dgdr.yaml -n $NAMESPACE
+```
+
+### Using AI Configurator (Fast Offline Profiling)
+
+For TensorRT-LLM, use AI Configurator for rapid profiling (~30 seconds):
+
+```yaml
+profilingConfig:
+  config:
+    sweep:
+      useAiConfigurator: true
+      aicSystem: h200_sxm
+      aicHfId: Qwen/Qwen3-32B
+      aicBackendVersion: "0.20.0"
+```
+
+### Direct Script Usage (Advanced)
+
+For advanced scenarios, run the profiler directly:
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend vllm \
+  --config path/to/disagg.yaml \
+  --model meta-llama/Llama-3-8B \
+  --ttft 200 --itl 15 \
+  --isl 3000 --osl 150
+```
+
+## Configuration
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `sla.isl` | - | Average input sequence length (tokens) |
+| `sla.osl` | - | Average output sequence length (tokens) |
+| `sla.ttft` | - | Target Time To First Token (milliseconds) |
+| `sla.itl` | - | Target Inter-Token Latency (milliseconds) |
+| `sweep.useAiConfigurator` | `false` | Use offline simulation instead of real profiling |
+| `hardware.minNumGpusPerEngine` | auto | Minimum GPUs per engine (auto-detected from model size) |
+| `hardware.maxNumGpusPerEngine` | 8 | Maximum GPUs per engine |
+
+## Profiling Methods
+
+| Method | Duration | Accuracy | GPU Required | Backends |
+|--------|----------|----------|--------------|----------|
+| Online (AIPerf) | 2-4 hours | Highest | Yes | All |
+| Offline (AI Configurator) | 20-30 seconds | Estimated | No | TensorRT-LLM |
+
+## Output
+
+The profiler generates:
+
+1. **Optimal Configuration**: Recommended TP sizes for prefill and decode engines
+2. **Performance Data**: Interpolation models for the SLA Planner
+3. **Generated DGD**: Complete deployment manifest with optimized settings
+
+Example recommendations:
+```text
+Suggested prefill TP:4 (TTFT 48.37 ms, throughput 15505.23 tokens/s/GPU)
+Suggested decode TP:4 (ITL 4.83 ms, throughput 51.22 tokens/s/GPU)
+```
+
+## Next Steps
+
+| Document | Description |
+|----------|-------------|
+| [Profiler Guide](profiler_guide.md) | Configuration, methods, and troubleshooting |
+| [Profiler Examples](profiler_examples.md) | Complete DGDR YAMLs, WebUI, script examples |
+| [SLA Planner Quick Start](/docs/planner/sla_planner_quickstart.md) | End-to-end deployment workflow |
+| [SLA Planner Architecture](/docs/planner/sla_planner.md) | How the Planner uses profiling data |
+
+```{toctree}
+:hidden:
+
+profiler_guide
+profiler_examples
+```

docs/components/profiler/profiler_examples.md

+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Profiler Examples
+
+Complete examples for profiling with DGDRs, the interactive WebUI, and direct script usage.
+
+## DGDR Examples
+
+### Dense Model: AIPerf on Real Engines
+
+Standard online profiling with real GPU measurements:
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: vllm-dense-online
+spec:
+  model: "Qwen/Qwen3-0.6B"
+  backend: vllm
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+    config:
+      sla:
+        isl: 3000
+        osl: 150
+        ttft: 200.0
+        itl: 20.0
+
+      hardware:
+        minNumGpusPerEngine: 1
+        maxNumGpusPerEngine: 8
+
+      sweep:
+        useAiConfigurator: false
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+
+  autoApply: true
+```
+
+### Dense Model: AI Configurator Simulation
+
+Fast offline profiling (~30 seconds, TensorRT-LLM only):
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: trtllm-aic-offline
+spec:
+  model: "Qwen/Qwen3-32B"
+  backend: trtllm
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.9.0"
+    config:
+      sla:
+        isl: 4000
+        osl: 500
+        ttft: 300.0
+        itl: 10.0
+
+      sweep:
+        useAiConfigurator: true
+        aicSystem: h200_sxm  # Also supports h100_sxm, b200_sxm, gb200_sxm, a100_sxm
+        aicHfId: Qwen/Qwen3-32B
+        aicBackendVersion: "0.20.0"
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.9.0"
+
+  autoApply: true
+```
+
+### MoE Model
+
+Multi-node MoE profiling with SGLang:
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: sglang-moe
+spec:
+  model: "deepseek-ai/DeepSeek-R1"
+  backend: sglang
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.9.0"
+    config:
+      sla:
+        isl: 2048
+        osl: 512
+        ttft: 300.0
+        itl: 25.0
+
+      hardware:
+        numGpusPerNode: 8
+        maxNumGpusPerEngine: 32
+
+      engine:
+        isMoeModel: true
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.9.0"
+
+  autoApply: true
+```
+
+### Using Existing DGD Config (ConfigMap)
+
+Reference a custom DGD configuration via ConfigMap:
+
+```bash
+# Create ConfigMap from your DGD config file
+kubectl create configmap deepseek-r1-config \
+  --from-file=/path/to/your/disagg.yaml \
+  --namespace $NAMESPACE \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: deepseek-r1
+spec:
+  model: deepseek-ai/DeepSeek-R1
+  backend: sglang
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.9.0"
+    configMapRef:
+      name: deepseek-r1-config
+      key: disagg.yaml
+    config:
+      sla:
+        isl: 4000
+        osl: 500
+        ttft: 300
+        itl: 10
+      sweep:
+        useAiConfigurator: true
+        aicSystem: h200_sxm
+        aicHfId: deepseek-ai/DeepSeek-V3
+        aicBackendVersion: "0.20.0"
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.9.0"
+
+  autoApply: true
+```
+
+## Interactive WebUI
+
+Launch an interactive configuration selection interface:
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend trtllm \
+  --config path/to/disagg.yaml \
+  --pick-with-webui \
+  --use-ai-configurator \
+  --model Qwen/Qwen3-32B-FP8 \
+  --aic-system h200_sxm \
+  --ttft 200 --itl 15
+```
+
+The WebUI launches on port 8000 by default (configurable with `--webui-port`).
+
+### Features
+
+- **Interactive Charts**: Visualize prefill TTFT, decode ITL, and GPU hours analysis with hover-to-highlight synchronization between charts and tables
+- **Pareto-Optimal Analysis**: The GPU Hours table shows pareto-optimal configurations balancing latency and throughput
+- **DGD Config Preview**: Click "Show Config" on any row to view the corresponding DynamoGraphDeployment YAML
+- **GPU Cost Estimation**: Toggle GPU cost display to convert GPU hours to cost ($/1000 requests)
+- **SLA Visualization**: Red dashed lines indicate your TTFT and ITL targets
+
+### Selection Methods
+
+1. **GPU Hours Table** (recommended): Click any row to select both prefill and decode configurations at once based on the pareto-optimal combination
+2. **Individual Selection**: Click one row in the Prefill table AND one row in the Decode table to manually choose each
+
+### Example DGD Config Output
+
+When you click "Show Config", you see a DynamoGraphDeployment configuration:
+
+```yaml
+# DynamoGraphDeployment Configuration
+# Prefill: 1 GPU(s), TP=1
+# Decode: 4 GPU(s), TP=4
+# Model: Qwen/Qwen3-32B-FP8
+# Backend: trtllm
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeployment
+spec:
+  services:
+    PrefillWorker:
+      subComponentType: prefill
+      replicas: 1
+      extraPodSpec:
+        mainContainer:
+          args:
+          - --tensor-parallel-size=1
+    DecodeWorker:
+      subComponentType: decode
+      replicas: 1
+      extraPodSpec:
+        mainContainer:
+          args:
+          - --tensor-parallel-size=4
+```
+
+Once you select a configuration, the full DGD CRD is saved as `config_with_planner.yaml`.
+
+## Direct Script Examples
+
+### Basic Profiling
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend vllm \
+  --config path/to/disagg.yaml \
+  --model meta-llama/Llama-3-8B \
+  --ttft 200 --itl 15 \
+  --isl 3000 --osl 150
+```
+
+### With GPU Constraints
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend sglang \
+  --config examples/backends/sglang/deploy/disagg.yaml \
+  --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
+  --ttft 200 --itl 15 \
+  --isl 3000 --osl 150 \
+  --min-num-gpus 2 \
+  --max-num-gpus 8
+```
+
+### AI Configurator (Offline)
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend trtllm \
+  --config path/to/disagg.yaml \
+  --use-ai-configurator \
+  --model Qwen/Qwen3-32B-FP8 \
+  --aic-system h200_sxm \
+  --ttft 200 --itl 15 \
+  --isl 4000 --osl 500
+```
+
+## SGLang Runtime Profiling
+
+Profile SGLang workers at runtime via HTTP endpoints:
+
+```bash
+# Start profiling
+curl -X POST http://localhost:9090/engine/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{"output_dir": "/tmp/profiler_output"}'
+
+# Run inference requests to generate profiling data...
+
+# Stop profiling
+curl -X POST http://localhost:9090/engine/stop_profile
+```
+
+A test script is provided at `examples/backends/sglang/test_sglang_profile.py`:
+
+```bash
+python examples/backends/sglang/test_sglang_profile.py
+```
+
+View traces using Chrome's `chrome://tracing`, [Perfetto UI](https://ui.perfetto.dev/), or TensorBoard.

docs/components/profiler/profiler_guide.md

+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Profiler Guide
+
+This guide covers deployment, configuration, integration, and troubleshooting for the Dynamo Profiler.
+
+## What is a DynamoGraphDeploymentRequest (DGDR)?
+
+A **DynamoGraphDeploymentRequest (DGDR)** is a Kubernetes Custom Resource that serves as the primary interface for users to request model deployments with specific performance and resource constraints. You specify:
+
+- **What** model you want to deploy (`model`)
+- **How** it should perform (SLA targets: `ttft`, `itl`)
+- **Where** it should run (optional GPU preferences)
+- **Which** backend to use (`backend`: vllm, sglang, or trtllm)
+- **Which** images to use (`profilingConfig.profilerImage`, `deploymentOverrides.workersImage`)
+
+The Dynamo Operator watches for DGDRs and automatically:
+1. Discovers available GPU resources in your cluster
+2. Runs profiling (online or offline) to find optimal configurations
+3. Generates an optimized DynamoGraphDeployment (DGD) configuration
+4. Deploys the DGD to your cluster
+
+**Relationship to DGD:**
+- **DGDR**: High-level "intent" - what you want deployed
+- **DGD**: Low-level "implementation" - how it's deployed
+
+## Support Matrix
+
+| Backend | Dense Models | MoE Models |
+|---------|-------------|------------|
+| vLLM | ✅ | 🚧 |
+| SGLang | ✅ | ✅ |
+| TensorRT-LLM | ✅ | 🚧 |
+
+The profiler sweeps over the following parallelization mappings for prefill and decode:
+
+| Model Architecture | Prefill Parallelization Mapping | Decode Parallelization Mapping |
+|---------|-------------|------------|
+| MLA+MoE (DeepseekV3ForCausalLM, DeepseekV32ForCausalLM) | TEP, DEP | TEP, DEP |
+| GQA+MoE (Qwen3MoeForCausalLM) | TP, TEP, DEP | TP, TEP, DEP |
+| Other Models | TP | TP |
+
+> [!NOTE]
+> Exact model x parallelization mapping support is dependent on the backend. The profiler does not guarantee that the recommended P/D engine configuration is supported and bug-free by the backend.
+
+## Deployment
+
+### Kubernetes Deployment (DGDR)
+
+The recommended deployment method is through DGDRs. Sample configurations are provided in `benchmarks/profiler/deploy/`:
+
+| Sample | Description |
+|--------|-------------|
+| `profile_sla_dgdr.yaml` | Standard online profiling with AIPerf |
+| `profile_sla_aic_dgdr.yaml` | Fast offline profiling with AI Configurator |
+| `profile_sla_moe_dgdr.yaml` | MoE model profiling (SGLang) |
+
+#### Container Images
+
+Each DGDR requires container images for profiling and deployment:
+
+- **`profilingConfig.profilerImage`** (Required): Container image for the profiling job. Must contain the profiler code and dependencies.
+- **`deploymentOverrides.workersImage`** (Optional): Container image for DGD worker components (frontend, workers, planner). If omitted, uses image from the base config file.
+
+```yaml
+spec:
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+```
+
+#### Quick Start: Deploy with DGDR
+
+**Step 1: Create Your DGDR**
+
+Use a sample configuration or create your own:
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: my-model-profiling
+spec:
+  model: "Qwen/Qwen3-0.6B"
+  backend: vllm
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+    config:
+      sla:
+        isl: 3000
+        osl: 150
+        ttft: 200.0
+        itl: 20.0
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+  autoApply: true
+```
+
+**Step 2: Apply the DGDR**
+
+```bash
+export NAMESPACE=your-namespace
+kubectl apply -f my-profiling-dgdr.yaml -n $NAMESPACE
+```
+
+**Step 3: Monitor Progress**
+
+```bash
+# View status
+kubectl get dgdr -n $NAMESPACE
+
+# Detailed status
+kubectl describe dgdr my-model-profiling -n $NAMESPACE
+
+# Watch profiling job logs
+kubectl logs -f job/profile-my-model-profiling -n $NAMESPACE
+```
+
+**DGDR Status States:**
+- `Pending`: Initial state, preparing to profile
+- `Profiling`: Running profiling job (20-30 seconds for AIC, 2-4 hours for online)
+- `Deploying`: Generating and applying DGD configuration
+- `Ready`: DGD successfully deployed and running
+- `Failed`: Error occurred (check events for details)
+
+**Step 4: Access Your Deployment**
+
+```bash
+# Find the frontend service
+kubectl get svc -n $NAMESPACE | grep frontend
+
+# Port-forward to access locally
+kubectl port-forward svc/<deployment>-frontend 8000:8000 -n $NAMESPACE
+
+# Test the endpoint
+curl http://localhost:8000/v1/models
+```
+
+> [!NOTE]
+> DGDRs are **immutable**. To update SLAs or configuration, delete the existing DGDR and create a new one.
+
+### Direct Script Execution
+
+For advanced use cases or local development:
+
+```bash
+python -m benchmarks.profiler.profile_sla \
+  --backend vllm \
+  --config path/to/disagg.yaml \
+  --model meta-llama/Llama-3-8B \
+  --ttft 200 --itl 15 \
+  --isl 3000 --osl 150 \
+  --min-num-gpus 1 \
+  --max-num-gpus 8
+```
+
+## Profiling Method
+
+The profiler follows a 5-step process:
+
+1. **Hardware Setup**: Uses defaults or user-specified hardware configuration. Optionally, cluster-scoped operators can enable automatic GPU discovery to detect specifications from cluster nodes.
+2. **Identify Sweep Ranges**: Automatically determine minimum and maximum number of GPUs per engine. Minimum is determined by the model size and GPU VRAM. Maximum is set to one node for dense models and 4 nodes for MoE models.
+3. **Parallelization Mapping Sweep**: Test performance of engines with different parallelization mappings using the input ISL and OSL.
+   - For dense models, test different TP sizes for both prefill and decode.
+   - For MoE models (SGLang), evaluate both TEP and DEP as candidates for prefill and decode.
+   - **Prefill**:
+     - TP/TEP: Measure TTFT with batch size = 1 (assuming ISL is long enough to saturate compute) without KV reuse.
+     - DEP: Attention uses data parallelism. Send a single burst with total concurrency `attention_dp_size × attn_dp_num_req_ratio` (defaults to 4) and compute the reported TTFT as `time_to_first_token.max / attn_dp_num_req_ratio` from the AIPerf summary of that burst.
+   ![Prefill Performance](../../images/h100_prefill_performance.png)
+   - **Decode**: Measure the ITL under different numbers of in-flight requests, from 1 to the maximum the KV cache can hold. To measure ITL without being affected by piggy-backed prefill requests, the script enables KV-reuse and warms up the engine by issuing the same prompts before measuring.
+   ![Decode Performance](../../images/h100_decode_performance.png)
+4. **Recommendation**: Select optimal parallelization mapping for prefill and decode that achieves the highest per-GPU throughput while adhering to the SLA on TTFT and ITL.
+5. **In-Depth Profiling on the Recommended P/D Engine**: Interpolate TTFT with ISL and ITL with active KV cache and decode context length for more accurate performance estimation.
+![ITL Interpolation](../../images/pd_interpolation.png)
+   - **Prefill**: Measures TTFT and throughput per GPU across different input lengths with batch size=1.
+   - **Decode**: Measures ITL and throughput per GPU under various KV cache loads and decode context lengths.
+
+### AIPerf on Real Engines
+
+Profiles your model by creating real test deployments in Kubernetes and measuring their performance.
+
+- **Duration**: 2-4 hours
+- **Accuracy**: Highest (real measurements)
+- **GPU Requirements**: Full access to test different parallelization mappings
+- **Backends**: vLLM, SGLang, TensorRT-LLM
+
+```yaml
+profilingConfig:
+  config:
+    sweep:
+      useAiConfigurator: false  # Default
+```
+
+### AI Configurator Simulation
+
+Uses performance simulation to rapidly estimate optimal configurations without running real deployments.
+
+- **Duration**: 20-30 seconds
+- **Accuracy**: Estimated (may have errors for unusual configurations)
+- **GPU Requirements**: None
+- **Backends**: TensorRT-LLM only (vLLM/SGLang coming soon)
+
+```yaml
+profilingConfig:
+  config:
+    sweep:
+      useAiConfigurator: true
+      aicSystem: h200_sxm
+      aicHfId: Qwen/Qwen3-32B
+      aicBackendVersion: "0.20.0"      # TRT-LLM version simulated by AIC
+```
+
+> [!NOTE]
+> `aicBackendVersion` specifies the TensorRT-LLM version that AI Configurator simulates. See the [AI Configurator supported features](https://github.com/ai-dynamo/aiconfigurator#supported-features) for available versions.
+
+**Currently supports:**
+- **Backends**: TensorRT-LLM (versions 0.20.0, 1.0.0rc3, 1.0.0rc6)
+- **Systems**: H100 SXM, H200 SXM, B200 SXM, GB200 SXM, A100 SXM
+- **Models**: Wide range including GPT, Llama, Mixtral, DeepSeek, Qwen, and more
+
+See [AI Configurator documentation](https://github.com/ai-dynamo/aiconfigurator#supported-features) for the full list.
+
+### Automatic GPU Discovery
+
+Cluster-scoped operators can optionally enable automatic GPU discovery:
+
+```yaml
+spec:
+  enableGpuDiscovery: true
+```
+
+This is only available with cluster-scoped operators (`namespaceRestriction.enabled=false`) as it requires cluster-wide node access permissions.
+
+## Configuration
+
+### DGDR Configuration Structure
+
+All profiler configuration goes under `spec.profilingConfig.config`:
+
+```yaml
+apiVersion: nvidia.com/v1alpha1
+kind: DynamoGraphDeploymentRequest
+metadata:
+  name: my-deployment
+spec:
+  model: "Qwen/Qwen3-0.6B"
+  backend: vllm
+
+  profilingConfig:
+    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+    configMapRef:                  # Optional: base DGD config
+      name: my-config
+      key: disagg.yaml
+
+    config:
+      sla: { ... }
+      hardware: { ... }
+      sweep: { ... }
+      planner: { ... }
+
+  deploymentOverrides:
+    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0"
+```
+
+### SLA Configuration (Required)
+
+```yaml
+sla:
+  isl: 3000      # Average input sequence length (tokens)
+  osl: 150       # Average output sequence length (tokens)
+  ttft: 200.0    # Target Time To First Token (milliseconds)
+  itl: 20.0      # Target Inter-Token Latency (milliseconds)
+```
+
+- **ISL/OSL**: Based on your expected traffic patterns
+- **TTFT**: First token latency target (lower = more GPUs needed, affects prefill engine)
+- **ITL**: Token generation latency target (lower = more GPUs needed, affects decode engine)
+- **Trade-offs**: Tighter SLAs require more GPU resources
+
+### Hardware Configuration (Optional)
+
+```yaml
+hardware:
+  minNumGpusPerEngine: 2      # Auto-determined from model size and VRAM if not provided
+  maxNumGpusPerEngine: 8      # Maximum GPUs to test
+  numGpusPerNode: 8           # GPUs per node (for multi-node MoE)
+  gpuType: h200_sxm           # GPU type hint (informational, auto-detected)
+```
+
+- **minNumGpusPerEngine**: Skip small TP sizes if your model is large
+- **maxNumGpusPerEngine**: Limit search space or work around constraints (e.g., [AIC attention heads](#ai-configurator-attention-head-constraint-error))
+- **numGpusPerNode**: Determine the upper bound of GPUs per node for dense models and configure Grove for multi-node MoE engines
+- **gpuType**: Informational only, auto-detected by the controller. For AI Configurator, use `aicSystem` in the [sweep configuration](#ai-configurator-configuration) instead
+
+> [!TIP]
+> If you don't specify hardware constraints, the controller auto-detects based on your model size and available cluster resources.
+
+### Sweep Configuration (Optional)
+
+```yaml
+sweep:
+  useAiConfigurator: false              # Use real profiling (default)
+  prefillInterpolationGranularity: 16   # Samples for prefill TTFT curve
+  decodeInterpolationGranularity: 6     # Samples for decode ITL curve
+```
+
+- **useAiConfigurator**: Set to `true` for 20-30 second profiling (TensorRT-LLM only)
+- **prefillInterpolationGranularity**: Samples for prefill TTFT curve (lower = faster but less accurate)
+- **decodeInterpolationGranularity**: Samples for decode ITL curve. Since ITL interpolation is 3D and takes longer, we default to fewer samples. Increasing this value may quadratically increase profiling time.
+
+### AI Configurator Configuration
+
+Required if `useAiConfigurator: true`:
+
+```yaml
+sweep:
+  useAiConfigurator: true
+  aicSystem: h200_sxm              # h100_sxm, h200_sxm, b200_sxm, gb200_sxm, a100_sxm
+  aicHfId: Qwen/Qwen3-32B         # HuggingFace model ID
+  aicBackendVersion: "0.20.0"      # TensorRT-LLM version
+```
+
+### Planner Configuration (Optional)
+
+Pass arguments to the SLA planner:
+
+```yaml
+planner:
+  planner_min_endpoint: 2                    # Minimum endpoints to maintain
+  planner_adjustment_interval: 60            # Adjustment interval (seconds)
+  planner_load_predictor: linear             # Load prediction method
+```
+
+> [!NOTE]
+> Planner arguments use `planner_` prefix. See [SLA Planner documentation](/docs/planner/sla_planner.md) for full list.
+
+### Model Cache PVC (Advanced)
+
+For large models, use a pre-populated PVC containing model weights instead of downloading from HuggingFace:
+
+```yaml
+deployment:
+  modelCache:
+    pvcName: "model-cache"
+    pvcPath: "hub/models--deepseek-ai--DeepSeek-R1"
+    mountPath: "/opt/model-cache"
+```
+
+Requirements:
+- The PVC must exist in the same namespace as the DGDR
+- The model weights must be accessible at `{mountPath}/{pvcPath}`
+
+### Engine Configuration (Auto-configured)
+
+The controller automatically injects these from high-level fields:
+
+```yaml
+# You specify:
+spec:
+  model: "Qwen/Qwen3-0.6B"
+  backend: vllm
+
+# Controller auto-injects:
+profilingConfig:
+  config:
+    deployment:
+      model: "Qwen/Qwen3-0.6B"
+    engine:
+      backend: vllm
+      config: /path/to/configmap
+```
+
+You should **not** manually set `deployment.model` or `engine.backend` in `profilingConfig.config`.
+
+### Using Existing DGD Configs (ConfigMap)
+
+Reference an existing DGD config via ConfigMap:
+
+```bash
+kubectl create configmap my-config \
+  --from-file=disagg.yaml=/path/to/your/disagg.yaml \
+  --namespace $NAMESPACE \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+
+```yaml
+profilingConfig:
+  configMapRef:
+    name: my-config
+    key: disagg.yaml
+```
+
+The profiler uses the DGD config as a **base template**, then optimizes it based on your SLA targets.
+
+### CLI Arguments
+
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `--backend` | string | - | Inference backend: vllm, sglang, trtllm |
+| `--config` | string | - | Path to DGD YAML config file |
+| `--model` | string | - | HuggingFace model ID |
+| `--ttft` | float | - | Target TTFT in milliseconds |
+| `--itl` | float | - | Target ITL in milliseconds |
+| `--isl` | int | - | Average input sequence length |
+| `--osl` | int | - | Average output sequence length |
+| `--min-num-gpus` | int | auto | Minimum GPUs per engine |
+| `--max-num-gpus` | int | 8 | Maximum GPUs per engine |
+| `--use-ai-configurator` | flag | false | Use offline AI Configurator |
+| `--pick-with-webui` | flag | false | Launch interactive WebUI |
+| `--webui-port` | int | 8000 | Port for WebUI |
+
+> [!NOTE]
+> CLI arguments map to DGDR config fields: `--min-num-gpus` = `hardware.minNumGpusPerEngine`, `--max-num-gpus` = `hardware.maxNumGpusPerEngine`, `--use-ai-configurator` = `sweep.useAiConfigurator`. See [DGDR Configuration Structure](#dgdr-configuration-structure) for all field mappings.
+
+## Integration
+
+### With SLA Planner
+
+The Profiler generates interpolation data that the SLA Planner uses for autoscaling decisions.
+
+**Prefill Interpolation** (`selected_prefill_interpolation/raw_data.npz`):
+- `prefill_isl`: 1D array of input sequence lengths tested
+- `prefill_ttft`: 1D array of TTFTs (ms) at each ISL
+- `prefill_thpt_per_gpu`: 1D array of throughput (tokens/s/GPU) at each ISL
+
+**Decode Interpolation** (`selected_decode_interpolation/raw_data.npz`):
+- `max_kv_tokens`: Total KV tokens capacity in decode engine
+- `x_kv_usage`: 1D array of active KV usage percentages [0, 1]
+- `y_context_length`: 1D array of average context lengths tested
+- `z_itl`: 1D array of ITLs (ms) at each (KV usage, context length) point
+- `z_thpt_per_gpu`: 1D array of throughput (tokens/s/GPU) at each point
+
+### With Dynamo Operator
+
+When using DGDR, the Dynamo Operator:
+
+1. Creates profiling jobs automatically
+2. Stores profiling data in ConfigMaps (`planner-profile-data`)
+3. Generates optimized DGD configurations
+4. Deploys the DGD with SLA Planner integration
+
+The generated DGD is tracked via labels:
+```yaml
+metadata:
+  labels:
+    dgdr.nvidia.com/name: my-deployment
+    dgdr.nvidia.com/namespace: your-namespace
+```
+
+### With Observability
+
+Monitor profiling jobs:
+
+```bash
+kubectl logs -f job/profile-<dgdr-name> -n $NAMESPACE
+kubectl describe dgdr <name> -n $NAMESPACE
+```
+
+## Advanced Topics
+
+### Manual Deployment Control
+
+Disable auto-deployment to review the generated DGD before applying:
+
+```yaml
+spec:
+  autoApply: false
+```
+
+Then manually extract and apply:
+
+```bash
+# Extract generated DGD from DGDR status
+kubectl get dgdr my-deployment -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' | kubectl apply -f -
+
+# Or save to file for review
+kubectl get dgdr my-deployment -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' > my-dgd.yaml
+```
+
+### Mocker Deployment
+
+Deploy a mocker deployment that simulates engines without GPUs:
+
+```yaml
+spec:
+  model: <model-name>
+  backend: trtllm
+  useMocker: true    # Deploy mocker instead of real backend
+  autoApply: true
+```
+
+Profiling still runs against the real backend to collect performance data. The mocker uses this data to simulate realistic timing behavior. Useful for large-scale experiments, testing Planner behavior, and validating configurations.
+
+### Accessing Profiling Artifacts
+
+By default, profiling data is stored in ConfigMaps. For detailed artifacts (plots, logs, raw data), attach a PVC:
+
+```yaml
+profilingConfig:
+  outputPVC: "dynamo-pvc"
+```
+
+**ConfigMaps (always created):**
+- `dgdr-output-<name>`: Generated DGD configuration
+- `planner-profile-data`: Profiling data for Planner (JSON)
+
+**PVC artifacts (optional):**
+- Performance plots (PNGs)
+- DGD configurations for each profiled deployment
+- AIPerf profiling artifacts
+- Raw profiling data (`.npz` files)
+- Profiler logs
+
+Access PVC results:
+```bash
+kubectl apply -f deploy/utils/manifests/pvc-access-pod.yaml -n $NAMESPACE
+kubectl wait --for=condition=Ready pod/pvc-access-pod -n $NAMESPACE --timeout=60s
+kubectl cp $NAMESPACE/pvc-access-pod:/data ./profiling-results
+kubectl delete pod pvc-access-pod -n $NAMESPACE
+```
+
+### Output Performance Plots
+
+The profiler generates plots to visualize performance data:
+
+**Parallelization Mapping Sweep Plots:**
+- `prefill_performance.png`: TTFT vs Parallelization Mapping size
+- `decode_performance.png`: ITL vs Parallelization Mapping size and in-flight requests
+
+**In-Depth Profiling Plots:**
+- `selected_prefill_interpolation/prefill_ttft_interpolation.png`: TTFT vs ISL
+- `selected_prefill_interpolation/prefill_throughput_interpolation.png`: Throughput vs ISL
+- `selected_decode_interpolation/decode_itl_interplation.png`: ITL vs KV usage and context length
+- `selected_decode_interpolation/decode_throughput_interpolation.png`: Throughput vs KV usage and context length
+
+## Runtime Profiling (SGLang)
+
+SGLang workers expose profiling endpoints for runtime performance analysis:
+
+```bash
+# Start profiling
+curl -X POST http://localhost:9090/engine/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{"output_dir": "/tmp/profiler_output"}'
+
+# Run inference requests...
+
+# Stop profiling
+curl -X POST http://localhost:9090/engine/stop_profile
+```
+
+View traces using Chrome's `chrome://tracing`, [Perfetto UI](https://ui.perfetto.dev/), or TensorBoard.
+
+## Troubleshooting
+
+### Profiling Takes Too Long
+
+**Solution 1**: Use AI Configurator for rapid profiling (TensorRT-LLM only):
+```yaml
+sweep:
+  useAiConfigurator: true
+```
+
+**Solution 2**: Reduce search space:
+```yaml
+hardware:
+  minNumGpusPerEngine: 4  # Skip TP1, TP2
+  maxNumGpusPerEngine: 8  # Don't test beyond TP8
+```
+
+### SLA Cannot Be Met
+
+**Symptoms**: Profiler reports no configuration meets targets
+
+**Solutions:**
+1. Relax SLA targets (increase TTFT/ITL)
+2. Add more GPU resources
+3. Try a different backend
+4. Use a smaller model
+
+### AI Configurator: Attention Head Constraint Error
+
+**Symptoms**: Profiling fails with error:
+```text
+AssertionError: num_heads <N> should be divisible by tp_size <M> and the division result should be >= 4
+```
+
+**Cause**: AI Configurator requires **≥4 attention heads per GPU**. Small models with few heads cannot use high TP sizes.
+
+**Affected Models:**
+- **Qwen3-0.6B** (16 heads): Max TP = 4
+- **GPT-2** (12 heads): Max TP = 3
+- Most models **<1B parameters**: May hit this constraint
+
+**Solution**: Limit `maxNumGpusPerEngine`:
+```yaml
+hardware:
+  maxNumGpusPerEngine: 4  # For Qwen3-0.6B (16 heads / 4 = max TP of 4)
+```
+
+**Calculate Max TP**: `max_tp = num_attention_heads / 4`
+
+> [!NOTE]
+> This is an AI Configurator limitation. Online profiling doesn't have this constraint.
+
+### Image Pull Errors
+
+**Symptoms**: `ErrImagePull` or `ImagePullBackOff`
+
+**Solution**: Ensure image pull secrets are configured:
+```bash
+kubectl create secret docker-registry nvcr-imagepullsecret \
+  --docker-server=nvcr.io \
+  --docker-username='$oauthtoken' \
+  --docker-password=<NGC_API_KEY> \
+  --namespace <your-namespace>
+```
+
+### Out of Memory During Profiling
+
+**Symptoms**: OOM errors in profiling jobs
+
+**Solutions:**
+1. Reduce `gpu_memory_utilization` in engine config
+2. Reduce `--max-context-length`
+3. Skip larger TP configurations
+4. Use fewer GPUs per test
+
+### Unsupported Parallelization Mapping in Backend
+
+**Symptoms**: Startup/runtime error in the backend (e.g., prime number of attention heads constraining TP to 1, or backend not supporting different TP sizes for prefill and decode).
+
+**Solutions:**
+1. Contact the backend to add support and bump backend version in Dynamo
+2. Constrain the max and min number of GPUs per engine to the supported range
+
+## See Also
+
+- [Profiler Examples](profiler_examples.md) - Complete DGDR YAML examples
+- [SLA Planner Quick Start](/docs/planner/sla_planner_quickstart.md) - End-to-end deployment workflow
+- [SLA Planner Architecture](/docs/planner/sla_planner.md) - How the Planner uses profiling data
+- [DGDR API Reference](/docs/kubernetes/api_reference.md) - DGDR specification
+- [Profiler Arguments Reference](/benchmarks/profiler/utils/profiler_argparse.py) - Full CLI reference

docs/index.rst

@@ -78,6 +78,7 @@ Quickstart
    Frontends <_sections/frontends>
    Router <router/README>
    Planner <planner/planner_intro>
+   Profiler <components/profiler/README>
    KVBM <kvbm/kvbm_intro>
 
 .. toctree::