---
id: pr-tensorrt-llm-11589
repo: NVIDIA/TensorRT-LLM
pr: 11589
title: '[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers'
author: nv-lschneider
date: '2026-04-27'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/11589
source_category: upstream-code
architectures:
- sm100
tags:
- allreduce
- benchmark_test
- cute dsl
- gemm_quant
- memory_primitives
- moe_routing
- userbuffers
techniques:
- autotuning
- fine-grained-quantization
- kernel-fusion
hardware_features:
- fp4
- fp8
- nvfp4
kernel_types:
- gemm
- memory
- moe
- norm
languages:
- cuda-cpp
- cute-dsl
captured_at: '2026-05-16'
status: merged
merge_sha: a518d394dd0d168a6c505a5420d7754fc4ca301a
inclusion_reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.
changed_paths:
- cpp/tensorrt_llm/nanobind/thop/bindings.cpp
- cpp/tensorrt_llm/thop/CMakeLists.txt
- cpp/tensorrt_llm/thop/allocateOutput.cpp
- cpp/tensorrt_llm/thop/allreduceOp.cpp
- cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp
- cpp/tensorrt_llm/thop/cublasScaledMM.cpp
- cpp/tensorrt_llm/thop/cublasScaledMM.h
- cpp/tensorrt_llm/thop/cudaNvfp4MM.cpp
- cpp/tensorrt_llm/thop/cudaScaledMM.cpp
- cpp/tensorrt_llm/thop/fp4Gemm.cpp
- cpp/tensorrt_llm/thop/fp8RowwiseGemm.cpp
- cpp/tensorrt_llm/thop/outputTensor.h
- cpp/tensorrt_llm/thop/weightOnlyQuantGemm.cpp
- cpp/tensorrt_llm/thop/weightOnlyQuantGemm.h
- tensorrt_llm/_torch/compilation/patterns/ar_residual_norm.py
- tensorrt_llm/_torch/custom_ops/__init__.py
- tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py
- tensorrt_llm/_torch/models/modeling_deepseekv3.py
- tensorrt_llm/_torch/modules/fused_moe/fused_moe_densegemm.py
- tests/microbenchmarks/all_reduce.py
- tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py
- tensorrt_llm/_torch/custom_ops/torch_custom_ops.py
- tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py
- tensorrt_llm/_torch/distributed/ops.py
- tensorrt_llm/_torch/models/modeling_glm.py
- tensorrt_llm/_torch/models/modeling_llama.py
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-11589
head_ref_oid: 08422a704ed7931c77246b54a819c95ab513daa7
---
# [TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [11589](https://github.com/NVIDIA/TensorRT-LLM/pull/11589)
- Decision: `Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.`
- Reason: Inspect scale layout, accumulator type, tile/schedule choice, epilogue fusion, and partial-tile guards before deriving a candidate.

## Changed Paths

- `cpp/tensorrt_llm/nanobind/thop/bindings.cpp`
- `cpp/tensorrt_llm/thop/CMakeLists.txt`
- `cpp/tensorrt_llm/thop/allocateOutput.cpp`
- `cpp/tensorrt_llm/thop/allreduceOp.cpp`
- `cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp`
- `cpp/tensorrt_llm/thop/cublasScaledMM.cpp`
- `cpp/tensorrt_llm/thop/cublasScaledMM.h`
- `cpp/tensorrt_llm/thop/cudaNvfp4MM.cpp`
- `cpp/tensorrt_llm/thop/cudaScaledMM.cpp`
- `cpp/tensorrt_llm/thop/fp4Gemm.cpp`
- `cpp/tensorrt_llm/thop/fp8RowwiseGemm.cpp`
- `cpp/tensorrt_llm/thop/outputTensor.h`
- `cpp/tensorrt_llm/thop/weightOnlyQuantGemm.cpp`
- `cpp/tensorrt_llm/thop/weightOnlyQuantGemm.h`
- `tensorrt_llm/_torch/compilation/patterns/ar_residual_norm.py`
- `tensorrt_llm/_torch/custom_ops/__init__.py`
- `tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py`
- `tensorrt_llm/_torch/models/modeling_deepseekv3.py`
- `tensorrt_llm/_torch/modules/fused_moe/fused_moe_densegemm.py`
- `tests/microbenchmarks/all_reduce.py`
- `tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py`
- `tensorrt_llm/_torch/custom_ops/torch_custom_ops.py`
- `tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py`
- `tensorrt_llm/_torch/distributed/ops.py`
- `tensorrt_llm/_torch/models/modeling_glm.py`
- `tensorrt_llm/_torch/models/modeling_llama.py`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-11589`
