---
id: pr-vllm-39917
repo: vllm-project/vllm
pr: 39917
title: '[Core] Replace routing replay with device cache and async D2H pipeline'
author: TomerBN-Nvidia
date: '2026-05-07'
url: https://github.com/vllm-project/vllm/pull/39917
source_category: upstream-code
architectures:
- sm100
tags:
- CUDA kernel
- FlashInfer
- MoE kernel
- memory_primitives
- moe_routing
- scheduler_autotune
- topk
techniques:
- autotuning
- cache-policy
- kernel-fusion
- pipeline-stages
- tile-scheduling
hardware_features: []
kernel_types:
- memory
- moe
- scheduler
languages:
- cuda-cpp
captured_at: '2026-05-16'
status: merged
merge_sha: 8189a15914ca48461acf106f126c58ef7e41c9ee
inclusion_reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.
changed_paths:
- docs/training/routed_experts_replay.md
- tests/model_executor/test_routed_experts_capture.py
- vllm/config/vllm.py
- vllm/entrypoints/openai/chat_completion/protocol.py
- vllm/entrypoints/openai/chat_completion/serving.py
- vllm/entrypoints/openai/completion/protocol.py
- vllm/entrypoints/openai/completion/serving.py
- vllm/model_executor/layers/fused_moe/layer.py
- vllm/model_executor/layers/fused_moe/routed_experts_capturer.py
- vllm/model_executor/layers/fused_moe/runner/moe_runner.py
- vllm/outputs.py
- vllm/v1/core/sched/scheduler.py
- vllm/v1/engine/output_processor.py
- vllm/v1/outputs.py
- vllm/v1/worker/gpu_model_runner.py
artifact_dir: evidence/pull-bundles/vllm/gh-39917
head_ref_oid: 11c7b316d122958083104031ed397c172d7537ce
---
# [Core] Replace routing replay with device cache and async D2H pipeline

- Repository: `vllm-project/vllm`
- PR: [39917](https://github.com/vllm-project/vllm/pull/39917)
- Decision: `Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.`
- Reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.

## Changed Paths

- `docs/training/routed_experts_replay.md`
- `tests/model_executor/test_routed_experts_capture.py`
- `vllm/config/vllm.py`
- `vllm/entrypoints/openai/chat_completion/protocol.py`
- `vllm/entrypoints/openai/chat_completion/serving.py`
- `vllm/entrypoints/openai/completion/protocol.py`
- `vllm/entrypoints/openai/completion/serving.py`
- `vllm/model_executor/layers/fused_moe/layer.py`
- `vllm/model_executor/layers/fused_moe/routed_experts_capturer.py`
- `vllm/model_executor/layers/fused_moe/runner/moe_runner.py`
- `vllm/outputs.py`
- `vllm/v1/core/sched/scheduler.py`
- `vllm/v1/engine/output_processor.py`
- `vllm/v1/outputs.py`
- `vllm/v1/worker/gpu_model_runner.py`

## Evidence Bundle

`evidence/pull-bundles/vllm/gh-39917`
