---
id: pr-tensorrt-llm-13033
repo: NVIDIA/TensorRT-LLM
pr: 13033
title: '[None][feat] Update rms_norm + fp4_qaunt kernel supporting more dim'
author: Wanli-Jiang
date: '2026-04-18'
url: https://github.com/NVIDIA/TensorRT-LLM/pull/13033
source_category: upstream-code
architectures:
- sm100
tags:
- benchmark kernel
- benchmark_test
- fp4
- gemm_quant
- norm_elementwise
techniques:
- fine-grained-quantization
- kernel-fusion
hardware_features:
- fp4
kernel_types:
- gemm
- moe
- norm
languages:
- cuda-cpp
captured_at: '2026-05-16'
status: merged
merge_sha: 74c57d12ca865a9724ff01a7a343fd9597e1df97
inclusion_reason: Look for fusion, vectorization, dtype conversion, and store-traffic reductions before changing math.
changed_paths:
- cpp/tensorrt_llm/kernels/fusedLayernormKernels/fp4_converter.cuh
- cpp/tensorrt_llm/kernels/fusedLayernormKernels/low_latency_layernorm.cuh
- cpp/tensorrt_llm/kernels/fusedLayernormKernels/ws_layernorm.cuh
- tensorrt_llm/_torch/models/modeling_nemotron_h.py
- tests/unittest/_torch/modules/test_fused_add_rms_norm_quant.py
artifact_dir: evidence/pull-bundles/tensorrt-llm/gh-13033
head_ref_oid: af4b29ac86b3c0348fe57ee0adfc535f8da95e60
---
# [None][feat] Update rms_norm + fp4_qaunt kernel supporting more dim

- Repository: `NVIDIA/TensorRT-LLM`
- PR: [13033](https://github.com/NVIDIA/TensorRT-LLM/pull/13033)
- Decision: `Look for fusion, vectorization, dtype conversion, and store-traffic reductions before changing math.`
- Reason: Look for fusion, vectorization, dtype conversion, and store-traffic reductions before changing math.

## Changed Paths

- `cpp/tensorrt_llm/kernels/fusedLayernormKernels/fp4_converter.cuh`
- `cpp/tensorrt_llm/kernels/fusedLayernormKernels/low_latency_layernorm.cuh`
- `cpp/tensorrt_llm/kernels/fusedLayernormKernels/ws_layernorm.cuh`
- `tensorrt_llm/_torch/models/modeling_nemotron_h.py`
- `tests/unittest/_torch/modules/test_fused_add_rms_norm_quant.py`

## Evidence Bundle

`evidence/pull-bundles/tensorrt-llm/gh-13033`
