---
id: pr-flashinfer-3152
repo: flashinfer-ai/flashinfer
pr: 3152
title: Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121
author: depaulmillz
date: '2026-05-05'
url: https://github.com/flashinfer-ai/flashinfer/pull/3152
source_category: upstream-code
architectures:
- sm100
tags:
- TMA
- arch_pipeline
- cutlass
- fp4
- gemm_quant
- mxfp4
- nvfp4
- sm120
techniques:
- async-copy
- fine-grained-quantization
- pipeline-stages
hardware_features:
- fp4
- fp8
- mxfp4
- nvfp4
- tma
kernel_types:
- gemm
- moe
- runtime
languages:
- cuda-cpp
- cute-dsl
captured_at: '2026-05-16'
status: merged
merge_sha: a11ce269d79647213d61fdd7099a7b57a3439cab
inclusion_reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.
changed_paths:
- 3rdparty/cutlass
- csrc/fp4_gemm_cutlass_sm120.cu
- csrc/fp4_gemm_cutlass_sm120.jinja
- csrc/group_gemm_mxfp4_groupwise_sm120.cu
- csrc/group_gemm_mxfp4_groupwise_sm120_kernel_inst.jinja
- csrc/group_gemm_nvfp4_groupwise_sm120.cu
- csrc/group_gemm_nvfp4_groupwise_sm120_kernel_inst.jinja
- csrc/group_gemm_sm120_binding.cu
- csrc/mxfp8_gemm_cutlass_sm120.jinja
- csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm_configs.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws.h
- flashinfer/gemm/gemm_base.py
- flashinfer/jit/gemm/core.py
- flashinfer/jit/gemm/cutlass/generate_kernels.py
- include/flashinfer/gemm/cutlass_gemm_configs.h
- include/flashinfer/gemm/fp4_gemm_cutlass_template_sm120.h
- include/flashinfer/gemm/fp4_gemm_template_sm120.h
- include/flashinfer/gemm/group_gemm_mxfp4_groupwise_sm120.cuh
- include/flashinfer/gemm/group_gemm_nvfp4_groupwise_sm120.cuh
- include/flashinfer/gemm/mxfp8_gemm_cutlass_template_sm120.h
- include/flashinfer/gemm/mxfp8_gemm_template_sm120.h
- tests/gemm/test_group_gemm_fp4.py
- tests/gemm/test_groupwise_scaled_gemm_mxfp4.py
- tests/gemm/test_mm_mxfp8_sm120.py
artifact_dir: evidence/pull-bundles/flashinfer/gh-3152
head_ref_oid: 010a611e1d2b7799f0e1507bd7c768c3ce8369d3
---
# Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121

- Repository: `flashinfer-ai/flashinfer`
- PR: [3152](https://github.com/flashinfer-ai/flashinfer/pull/3152)
- Decision: `Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.`
- Reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.

## Changed Paths

- `3rdparty/cutlass`
- `csrc/fp4_gemm_cutlass_sm120.cu`
- `csrc/fp4_gemm_cutlass_sm120.jinja`
- `csrc/group_gemm_mxfp4_groupwise_sm120.cu`
- `csrc/group_gemm_mxfp4_groupwise_sm120_kernel_inst.jinja`
- `csrc/group_gemm_nvfp4_groupwise_sm120.cu`
- `csrc/group_gemm_nvfp4_groupwise_sm120_kernel_inst.jinja`
- `csrc/group_gemm_sm120_binding.cu`
- `csrc/mxfp8_gemm_cutlass_sm120.jinja`
- `csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm_configs.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws.h`
- `flashinfer/gemm/gemm_base.py`
- `flashinfer/jit/gemm/core.py`
- `flashinfer/jit/gemm/cutlass/generate_kernels.py`
- `include/flashinfer/gemm/cutlass_gemm_configs.h`
- `include/flashinfer/gemm/fp4_gemm_cutlass_template_sm120.h`
- `include/flashinfer/gemm/fp4_gemm_template_sm120.h`
- `include/flashinfer/gemm/group_gemm_mxfp4_groupwise_sm120.cuh`
- `include/flashinfer/gemm/group_gemm_nvfp4_groupwise_sm120.cuh`
- `include/flashinfer/gemm/mxfp8_gemm_cutlass_template_sm120.h`
- `include/flashinfer/gemm/mxfp8_gemm_template_sm120.h`
- `tests/gemm/test_group_gemm_fp4.py`
- `tests/gemm/test_groupwise_scaled_gemm_mxfp4.py`
- `tests/gemm/test_mm_mxfp8_sm120.py`

## Evidence Bundle

`evidence/pull-bundles/flashinfer/gh-3152`
