---
id: pr-tensorrt-llm-12136
repo: NVIDIA/TensorRT-LLM
pr: 12136
title: '[None][feat] Add DWDP (Distributed Weight Data Parallelism) support for MoE inference'
author: tianyuz-nv
date: '2026-04-02'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/12136
source_category: upstream-code
architectures:
- sm100
tags:
- arch_pipeline
- benchmark kernel
- benchmark_test
- cute dsl
- gemm_quant
- moe_routing
techniques:
- fine-grained-quantization
- kernel-fusion
- pipeline-stages
hardware_features:
- blackwell
kernel_types:
- gemm
- moe
- runtime
languages:
- cuda-cpp
- cute-dsl
captured_at: '2026-05-16'
status: merged
merge_sha: e92ee4fe550783a5f3a51cd690d89d1fe18ea30b
inclusion_reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.
changed_paths:
- cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp
- examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm
- examples/disaggregated/slurm/benchmark/start_worker_dwdp.sh
- examples/disaggregated/slurm/benchmark/submit.py
- examples/disaggregated/slurm/benchmark/submit_dwdp.py
- tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py
- tensorrt_llm/_torch/cute_dsl_kernels/blackwell/blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py
- tensorrt_llm/_torch/cute_dsl_kernels/blackwell/blockscaled_contiguous_grouped_gemm_finalize_fusion.py
- tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py
- tensorrt_llm/_torch/modules/fused_moe/fused_moe_cute_dsl.py
- tensorrt_llm/_torch/modules/fused_moe/interface.py
- tensorrt_llm/_torch/pyexecutor/_util.py
- tensorrt_llm/_torch/pyexecutor/dwdp.py
- tensorrt_llm/_torch/pyexecutor/py_executor.py
- tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
- tensorrt_llm/llmapi/llm_args.py
- tests/integration/defs/accuracy/test_dwdp_disaggregated_serving.py
- tests/integration/test_lists/qa/llm_function_core.txt
- tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml
- tests/scripts/cute_dsl_kernels/run_blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py
- tests/scripts/cute_dsl_kernels/run_blockscaled_contiguous_grouped_gemm_finalize_fusion.py
- tests/unittest/_torch/thop/parallel/test_cute_dsl_moe.py
- tests/unittest/api_stability/references/llm.yaml
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-12136
head_ref_oid: be1248293e7e337dc4a7ff544038e40d36701e38
---
# [None][feat] Add DWDP (Distributed Weight Data Parallelism) support for MoE inference

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [12136](https://github.com/NVIDIA/TensorRT-LLM/pull/12136)
- Decision: `Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.`
- Reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.

## Changed Paths

- `cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp`
- `examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm`
- `examples/disaggregated/slurm/benchmark/start_worker_dwdp.sh`
- `examples/disaggregated/slurm/benchmark/submit.py`
- `examples/disaggregated/slurm/benchmark/submit_dwdp.py`
- `tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py`
- `tensorrt_llm/_torch/cute_dsl_kernels/blackwell/blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py`
- `tensorrt_llm/_torch/cute_dsl_kernels/blackwell/blockscaled_contiguous_grouped_gemm_finalize_fusion.py`
- `tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py`
- `tensorrt_llm/_torch/modules/fused_moe/fused_moe_cute_dsl.py`
- `tensorrt_llm/_torch/modules/fused_moe/interface.py`
- `tensorrt_llm/_torch/pyexecutor/_util.py`
- `tensorrt_llm/_torch/pyexecutor/dwdp.py`
- `tensorrt_llm/_torch/pyexecutor/py_executor.py`
- `tensorrt_llm/_torch/pyexecutor/py_executor_creator.py`
- `tensorrt_llm/llmapi/llm_args.py`
- `tests/integration/defs/accuracy/test_dwdp_disaggregated_serving.py`
- `tests/integration/test_lists/qa/llm_function_core.txt`
- `tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml`
- `tests/scripts/cute_dsl_kernels/run_blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py`
- `tests/scripts/cute_dsl_kernels/run_blockscaled_contiguous_grouped_gemm_finalize_fusion.py`
- `tests/unittest/_torch/thop/parallel/test_cute_dsl_moe.py`
- `tests/unittest/api_stability/references/llm.yaml`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-12136`
