---
id: pr-flashinfer-1294
repo: flashinfer-ai/flashinfer
pr: 1294
title: Update cutlass fp4 moe kernels
author: wenscarl
date: '2025-07-23'
url: https://github.com/flashinfer-ai/flashinfer/pull/1294
source_category: upstream-code
architectures:
- sm100
tags:
- fp4
- fp8
- gemm
- kernel-fusion
- moe
- quantization
- tma
- warp-specialization
techniques:
- kernel-fusion
- warp-specialization
hardware_features:
- fp4
- fp8
- tma
kernel_types:
- gemm
- moe
- quantization
languages:
- cuda-cpp
captured_at: '2026-04-27'
status: merged
merge_sha: 9c609f050c7b753199b845c3efae5dcb956d0b17
inclusion_reason: kernel file changes
changed_paths:
- benchmarks/bench_cutlass_fused_moe.py
- csrc/fused_moe/cutlass_backend/cutlass_fused_moe_instantiation.cu
- csrc/fused_moe/cutlass_backend/cutlass_fused_moe_kernels.cuh
- csrc/fused_moe/cutlass_backend/flashinfer_cutlass_fused_moe_sm100_ops.cu
- csrc/nv_internal/cpp/kernels/quantization.cu
- csrc/nv_internal/include/tensorrt_llm/common/NvInferRuntime.h
- csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/moe_cutlass_kernel.h
- csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm_configs.h
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M128_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M128_BSTrue_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M256_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M256_BSTrue_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M64_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M128_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M16_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M32_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M64_BSFalse_MixedFalse.generated.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm_stub.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/common.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/cutlass_kernel_selector.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_gemm_kernels.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/fused_moe_gemm_launcher_sm80.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/fused_moe_gemm_launcher_sm80.inl
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_bf16.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_fp8.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_uint4.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_uint8.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_fp16.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_uint4.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_uint8.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp32_fp32.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp4_fp4.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_fp4.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_fp8.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_uint4.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws_mixed_dtype.h
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_tma_warp_specialized_input.cu
- csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_tma_warp_specialized_traits.h
- csrc/nv_internal/tensorrt_llm/kernels/internal_cutlass_kernels/src/moe_gemm/launchers/moe_gemm_tma_ws_launcher.h
- csrc/nv_internal/tensorrt_llm/kernels/lora/lora.cpp
- csrc/nv_internal/tensorrt_llm/kernels/preQuantScaleKernel.cu
- csrc/nv_internal/tensorrt_llm/kernels/preQuantScaleKernel.h
- csrc/nv_internal/tensorrt_llm/kernels/quantization.cuh
- csrc/nv_internal/tensorrt_llm/kernels/quantization.h
- csrc/nv_internal/tensorrt_llm/thop/fp4Op.cpp
- csrc/nv_internal/tensorrt_llm/thop/fp4Quantize.cpp
- csrc/nv_internal/tensorrt_llm/thop/fp4Quantize.h
- flashinfer/__init__.py
- flashinfer/autotuner.py
- flashinfer/fp4_quantization.py
- flashinfer/fused_moe.py
- flashinfer/fused_moe_utils.py
- flashinfer/tllm_utils.py
- flashinfer/utils.py
- tests/test_fp4_quantize.py
- tests/test_trtllm_cutlass_fused_moe.py
head_ref_oid: e22bdbd327c91c22fa7ae6935fe662ca3019ede0
artifact_dir: evidence/pull-bundles/flashinfer/gh-1294
---
# Update cutlass fp4 moe kernels

- Repository: `flashinfer-ai/flashinfer`
- PR: [1294](https://github.com/flashinfer-ai/flashinfer/pull/1294)
- Decision: `kernel file changes`
- Reason: kernel file changes

## Changed Paths

- `benchmarks/bench_cutlass_fused_moe.py`
- `csrc/fused_moe/cutlass_backend/cutlass_fused_moe_instantiation.cu`
- `csrc/fused_moe/cutlass_backend/cutlass_fused_moe_kernels.cuh`
- `csrc/fused_moe/cutlass_backend/flashinfer_cutlass_fused_moe_sm100_ops.cu`
- `csrc/nv_internal/cpp/kernels/quantization.cu`
- `csrc/nv_internal/include/tensorrt_llm/common/NvInferRuntime.h`
- `csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/moe_cutlass_kernel.h`
- `csrc/nv_internal/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm_configs.h`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M128_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M128_BSTrue_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M256_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M256_BSTrue_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm100_M64_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M128_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M16_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M32_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/cutlass_instantiations/gemm_grouped/cutlass_kernel_file_GemmKind.Grouped_sm80_M64_BSFalse_MixedFalse.generated.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm_stub.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/common.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/cutlass_kernel_selector.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_gemm_kernels.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/fused_moe_gemm_launcher_sm80.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/fused_moe_gemm_launcher_sm80.inl`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_bf16.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_fp8.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_uint4.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_bf16_uint8.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_fp16.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_uint4.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp16_uint8.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp32_fp32.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp4_fp4.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_fp4.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_fp8.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_kernels_fp8_uint4.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch_tma_ws_mixed_dtype.h`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_tma_warp_specialized_input.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_tma_warp_specialized_traits.h`
- `csrc/nv_internal/tensorrt_llm/kernels/internal_cutlass_kernels/src/moe_gemm/launchers/moe_gemm_tma_ws_launcher.h`
- `csrc/nv_internal/tensorrt_llm/kernels/lora/lora.cpp`
- `csrc/nv_internal/tensorrt_llm/kernels/preQuantScaleKernel.cu`
- `csrc/nv_internal/tensorrt_llm/kernels/preQuantScaleKernel.h`
- `csrc/nv_internal/tensorrt_llm/kernels/quantization.cuh`
- `csrc/nv_internal/tensorrt_llm/kernels/quantization.h`
- `csrc/nv_internal/tensorrt_llm/thop/fp4Op.cpp`
- `csrc/nv_internal/tensorrt_llm/thop/fp4Quantize.cpp`
- `csrc/nv_internal/tensorrt_llm/thop/fp4Quantize.h`
- `flashinfer/__init__.py`
- `flashinfer/autotuner.py`
- `flashinfer/fp4_quantization.py`
- `flashinfer/fused_moe.py`
- `flashinfer/fused_moe_utils.py`
- `flashinfer/tllm_utils.py`
- `flashinfer/utils.py`
- `tests/test_fp4_quantize.py`
- `tests/test_trtllm_cutlass_fused_moe.py`

## Evidence Bundle

`evidence/pull-bundles/flashinfer/gh-1294`
