---
id: pr-tensorrt-llm-9838
repo: NVIDIA/TensorRT-LLM
pr: 9838
title: '[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE'
author: yumin066
date: '2026-01-06'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/9838
source_category: upstream-code
architectures:
- sm90
tags:
- benchmark_test
- gemm_quant
- moe_routing
- sm90
techniques:
- async-copy
- epilogue-fusion
- fine-grained-quantization
- kernel-fusion
hardware_features:
- fp8
- hopper
- tma
kernel_types:
- gemm
- moe
languages:
- cuda-cpp
- cute-dsl
captured_at: '2026-05-16'
status: merged
merge_sha: 9cae7277ea9cc41242ede456ea1955902962f971
inclusion_reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.
changed_paths:
- cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws_mixed_dtype.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu
- cpp/tensorrt_llm/kernels/cutlass_kernels/python/generate_kernels.py
- cpp/tensorrt_llm/thop/moeOp.cpp
- cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-9838
head_ref_oid: d2384f1f61a7884975a97d57fcaecea158faa8b9
---
# [https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [9838](https://github.com/NVIDIA/TensorRT-LLM/pull/9838)
- Decision: `Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.`
- Reason: Separate routing, permutation, top-k, and expert GEMM costs; keep tail-token and expert-layout tests attached to the idea.

## Changed Paths

- `cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws_mixed_dtype.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/python/generate_kernels.py`
- `cpp/tensorrt_llm/thop/moeOp.cpp`
- `cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-9838`
