---
id: pr-vllm-40408
repo: vllm-project/vllm
pr: 40408
title: '[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement'
author: yewentao256
date: '2026-05-11'
url: https://github.com/vllm-project/vllm/pull/40408
source_category: upstream-code
architectures:
- sm100
- sm90
tags:
- DeepGEMM
- arch_pipeline
- benchmark_test
- cutlass
- fp8
- gemm_quant
techniques:
- fine-grained-quantization
- pipeline-stages
hardware_features:
- blackwell
- fp8
- hopper
kernel_types:
- attention
- gemm
- runtime
languages:
- cuda-cpp
- cute-dsl
- triton
captured_at: '2026-05-16'
status: merged
merge_sha: 0d453e23364701f6fe2f20deafbb372b28714a86
inclusion_reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.
changed_paths:
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8.cu
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8_dispatch.cuh
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8.cu
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8.cu
- csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8_dispatch.cuh
- csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x.cu
- csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x_sm89_fp8_dispatch.cuh
- tests/v1/determinism/test_cutlass_batch_invariance.py
- vllm/model_executor/layers/quantization/fp8.py
- vllm/model_executor/layers/quantization/online/fp8.py
artifact_dir: evidence/pull-bundles/vllm/gh-40408
head_ref_oid: df69311190eb52bb9c2f14a5855e2a206bb088ee
---
# [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement

- Repository: `vllm-project/vllm`
- PR: [40408](https://github.com/vllm-project/vllm/pull/40408)
- Decision: `Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.`
- Reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.

## Changed Paths

- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8.cu`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8_dispatch.cuh`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8.cu`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8.cu`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8_dispatch.cuh`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x.cu`
- `csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x_sm89_fp8_dispatch.cuh`
- `tests/v1/determinism/test_cutlass_batch_invariance.py`
- `vllm/model_executor/layers/quantization/fp8.py`
- `vllm/model_executor/layers/quantization/online/fp8.py`

## Evidence Bundle

`evidence/pull-bundles/vllm/gh-40408`
