---
id: pr-tensorrt-llm-12470
repo: NVIDIA/TensorRT-LLM
pr: 12470
title: '[None][feat] Support sparse mqa/gqa attention'
author: heyuhhh
date: '2026-04-19'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/12470
source_category: upstream-code
architectures:
- sm100
tags:
- DSA
- MLA
- attention kernel
- attention_kv
- benchmark kernel
- paged MQA
techniques:
- kernel-fusion
- persistent-kernels
hardware_features:
- blackwell
kernel_types:
- attention
- decode
- mla
languages:
- cuda-cpp
captured_at: '2026-05-16'
status: merged
merge_sha: cf9963f19550156e4bf7d3d8d8d90bd0b7a8a4a8
inclusion_reason: Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.
changed_paths:
- cpp/tensorrt_llm/common/attentionOp.cpp
- cpp/tensorrt_llm/common/attentionOp.h
- cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h
- cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplCommon.h
- cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h
- cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp
- cpp/tensorrt_llm/kernels/sparseAttentionKernels.h
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin.cpp
- cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin.cpp
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-12470
head_ref_oid: 95036b7a3098cfe8a1d00ac8c54436b30c362756
---
# [None][feat] Support sparse mqa/gqa attention

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [12470](https://github.com/NVIDIA/TensorRT-LLM/pull/12470)
- Decision: `Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.`
- Reason: Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.

## Changed Paths

- `cpp/tensorrt_llm/common/attentionOp.cpp`
- `cpp/tensorrt_llm/common/attentionOp.h`
- `cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h`
- `cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplCommon.h`
- `cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h`
- `cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp`
- `cpp/tensorrt_llm/kernels/sparseAttentionKernels.h`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin.cpp`
- `cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin.cpp`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-12470`
