---
id: pr-tensorrt-llm-8501
repo: NVIDIA/TensorRT-LLM
pr: 8501
title: '[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP'
author: jinyangyuan-nvidia
date: '2025-10-27'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/8501
source_category: upstream-code
architectures:
- sm100
tags:
- benchmark_test
- gemm_quant
- moe_routing
- userbuffers
techniques:
- fine-grained-quantization
- kernel-fusion
hardware_features:
- fp8
kernel_types:
- attention
- gemm
- moe
languages:
- cuda-cpp
- cute-dsl
captured_at: '2026-05-16'
status: merged
merge_sha: 0a0f93d4a8f577f24ea2b8a0cf28d976c3502725
inclusion_reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.
changed_paths:
- cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.cu
- cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h
- cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu
- cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h
- cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.cpp
- cpp/tensorrt_llm/thop/fp8BlockScalingGemm.cpp
- cpp/tensorrt_llm/thop/moeOp.cpp
- cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu
- tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
- tensorrt_llm/_torch/modules/fused_moe/ops/moe_op_cutlass.py
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-8501
head_ref_oid: 55bc09c9554b4da440907d62ef79257325e63b94
---
# [None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [8501](https://github.com/NVIDIA/TensorRT-LLM/pull/8501)
- Decision: `Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.`
- Reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.

## Changed Paths

- `cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.cu`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h`
- `cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu`
- `cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h`
- `cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.cpp`
- `cpp/tensorrt_llm/thop/fp8BlockScalingGemm.cpp`
- `cpp/tensorrt_llm/thop/moeOp.cpp`
- `cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu`
- `tensorrt_llm/_torch/custom_ops/torch_custom_ops.py`
- `tensorrt_llm/_torch/modules/fused_moe/ops/moe_op_cutlass.py`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-8501`
