docs(router): add tuning guideline for output block tracking (#6911)

Signed-off-by: PeaBrane <yanrpei@gmail.com>

benchmarks/router/common.py

@@ -135,6 +135,12 @@ def add_synthesis_args(parser):
         default=None,
         help="Maximum output sequence length - clips values above this threshold (default: None, no clipping)",
     )
+    parser.add_argument(
+        "--osl-multiplier",
+        type=float,
+        default=1.0,
+        help="Multiplier for output sequence lengths (default: 1.0)",
+    )
     parser.add_argument(
         "--block-size",
         type=int,
@@ -220,6 +226,7 @@ def prepare_trace_dataset(args, output_dir, logger):
         or args.prefix_len_multiplier != 1.0
         or args.prefix_root_multiplier != 1
         or args.prompt_len_multiplier != 1.0
+        or args.osl_multiplier != 1.0
         or args.max_isl is not None
         or args.min_isl is not None
         or args.min_osl is not None
@@ -269,6 +276,7 @@ def prepare_trace_dataset(args, output_dir, logger):
     logger.info(f"  Prefix len multiplier: {args.prefix_len_multiplier}")
     logger.info(f"  Prefix root multiplier: {args.prefix_root_multiplier}")
     logger.info(f"  Prompt len multiplier: {args.prompt_len_multiplier}")
+    logger.info(f"  OSL multiplier: {args.osl_multiplier}")
     logger.info(
         f"  Max ISL: {args.max_isl if args.max_isl else 'no limit'} (filtering)"
     )
@@ -292,6 +300,7 @@ def prepare_trace_dataset(args, output_dir, logger):
         prefix_len_multiplier=args.prefix_len_multiplier,
         prefix_root_multiplier=args.prefix_root_multiplier,
         prompt_len_multiplier=args.prompt_len_multiplier,
+        osl_multiplier=args.osl_multiplier,
     )
 
     if args.num_requests is None:

docs/components/router/router-guide.md

@@ -190,7 +190,7 @@ The main KV-aware routing arguments (frontend uses the same `--router-*` flag na
 
 - `--no-router-track-active-blocks`: Disables tracking of active blocks (blocks being used for ongoing generation/decode phases). By default, the router tracks active blocks for load balancing. Disable this when routing to workers that only perform prefill (no decode phase), as tracking decode load is not relevant. This reduces router overhead and simplifies state management.
 
-- `--router-track-output-blocks`: Enables tracking of output blocks during generation (default: disabled). When enabled, the router adds placeholder blocks as tokens are generated and applies fractional decay based on progress toward the expected output sequence length (`agent_hints.osl` in nvext). This improves load balancing accuracy for long-running generation requests by accounting for output-side KV cache growth.
+- `--router-track-output-blocks`: **(Experimental)** Enables tracking of output blocks during generation (default: disabled). When enabled, the router adds placeholder blocks as tokens are generated and applies fractional decay based on progress toward the expected output sequence length (`agent_hints.osl` in nvext). This improves load balancing accuracy for long-running generation requests by accounting for output-side KV cache growth.
 
 - `--no-router-assume-kv-reuse`: When tracking active blocks, disables the assumption of KV cache reuse. By default (`router_assume_kv_reuse=true`), the router computes actual block hashes for sequence tracking to deduplicate blocks and optimize load balancing. When disabled via this flag, the router generates random hashes for sequence blocks, treating each request's blocks as unique. This is useful in disaggregated setups where prefill transfers blocks to decode workers that may already have those blocks cached, but the engine cannot coordinate transfers to avoid duplication. Without this flag, the router's load balancing heuristics would undercount decode blocks when duplicates exist.
 
@@ -218,6 +218,8 @@ Use `--no-router-kv-events` when you are not confident that your backend engine
 
 Use `--no-router-assume-kv-reuse` in disaggregated setups where the decode worker does not reuse transferred KV cache blocks. By default the router assumes KV blocks transferred from prefill to decode will be deduplicated on the decode side, but vLLM and SGLang decode workers currently do not support this — only TensorRT-LLM does. Without this flag, the router undercounts decode blocks when duplicates exist, leading to inaccurate load estimates.
 
+Use `--router-track-output-blocks` **(experimental)** when your workload is output-heavy and you want the router to account for output-side KV cache growth in load balancing. This is useful in two scenarios: (1) workloads with long output sequences and little multi-turn reuse, where output blocks dominate the KV cache footprint; (2) agentic schedulers (e.g. NAT or other LLM routers) that can accurately predict the expected output sequence length per request. When enabled, the router adds placeholder blocks as tokens are generated. If you additionally pass `nvext.agent_hints.osl` (expected output sequence length in tokens) per request, the router applies fractional decay to output blocks — each output block's weight starts at 1.0 and decays linearly toward 0.0 as generation approaches the expected OSL. This lets the router predict that a request nearing completion will soon free its blocks, effectively modeling the future load trajectory rather than just the current snapshot. Without `osl`, output blocks are added at full weight with no decay. The flag requires `--router-track-active-blocks` (the default).
+
 Set `--router-queue-threshold` (e.g. `1.5`) to enable backpressure under very high concurrency workloads. When set, the router holds incoming requests in a priority queue while all workers exceed the given fraction of `max_num_batched_tokens`, releasing them as capacity frees up. This defers the routing decision so it is made with the freshest load metrics, rather than dispatching into an already-saturated system. It also enables priority scheduling via `nvext.agent_hints.latency_sensitivity`.
 
 ### Prometheus Metrics