---
id: pr-vllm-16113
repo: vllm-project/vllm
pr: 16113
title: Upstream Llama4 Support to Main
author: houseroad
date: '2025-04-07'
url: https://github.com/vllm-project/vllm/pull/16113
source_category: upstream-code
architectures:
- sm100
tags:
- attention
- decode
- fp8
- kernel-fusion
- moe
- pipeline-stages
- quantization
techniques:
- kernel-fusion
- pipeline-stages
hardware_features:
- fp8
kernel_types:
- attention
- decode
- moe
- quantization
languages:
- triton
captured_at: '2026-04-27'
status: merged
merge_sha: 55dcce91df150f576c28520d987eaf1498fcb0bd
inclusion_reason: kernel file changes
changed_paths:
- .buildkite/test-pipeline.yaml
- benchmarks/kernels/benchmark_moe.py
- docs/source/models/supported_models.md
- examples/offline_inference/audio_language.py
- examples/offline_inference/vision_language.py
- examples/offline_inference/vision_language_multi_image.py
- requirements/common.txt
- requirements/test.in
- requirements/test.txt
- tests/models/decoder_only/audio_language/test_ultravox.py
- tests/models/decoder_only/vision_language/test_models.py
- tests/models/decoder_only/vision_language/test_phi3v.py
- tests/models/decoder_only/vision_language/test_pixtral.py
- tests/models/multimodal/processing/test_common.py
- tests/models/multimodal/processing/test_llama4.py
- tests/models/registry.py
- tests/models/test_initialization.py
- vllm/config.py
- vllm/entrypoints/chat_utils.py
- vllm/model_executor/layers/fused_moe/configs/E=16,N=1024,device_name=AMD_Instinct_MI300X.json
- vllm/model_executor/layers/fused_moe/cutlass_moe.py
- vllm/model_executor/layers/fused_moe/fused_moe.py
- vllm/model_executor/layers/fused_moe/layer.py
- vllm/model_executor/layers/layernorm.py
- vllm/model_executor/layers/quantization/awq_marlin.py
- vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
- vllm/model_executor/layers/quantization/experts_int8.py
- vllm/model_executor/layers/quantization/fp8.py
- vllm/model_executor/layers/quantization/gguf.py
- vllm/model_executor/layers/quantization/gptq_marlin.py
- vllm/model_executor/layers/quantization/moe_wna16.py
- vllm/model_executor/layers/quantization/quark/quark_moe.py
- vllm/model_executor/layers/rotary_embedding.py
- vllm/model_executor/model_loader/loader.py
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/llama4.py
- vllm/model_executor/models/mllama4.py
- vllm/model_executor/models/registry.py
- vllm/model_executor/models/telechat2.py
- vllm/model_executor/models/teleflm.py
- vllm/v1/attention/backends/flash_attn.py
- vllm/v1/attention/backends/triton_attn.py
- vllm/v1/worker/gpu_model_runner.py
head_ref_oid: 834d3e0b57e14a3ad6bd177a2cce486a24ee509e
artifact_dir: evidence/pull-bundles/vllm/gh-16113
---
# Upstream Llama4 Support to Main

- Repository: `vllm-project/vllm`
- PR: [16113](https://github.com/vllm-project/vllm/pull/16113)
- Decision: `kernel file changes`
- Reason: kernel file changes

## Changed Paths

- `.buildkite/test-pipeline.yaml`
- `benchmarks/kernels/benchmark_moe.py`
- `docs/source/models/supported_models.md`
- `examples/offline_inference/audio_language.py`
- `examples/offline_inference/vision_language.py`
- `examples/offline_inference/vision_language_multi_image.py`
- `requirements/common.txt`
- `requirements/test.in`
- `requirements/test.txt`
- `tests/models/decoder_only/audio_language/test_ultravox.py`
- `tests/models/decoder_only/vision_language/test_models.py`
- `tests/models/decoder_only/vision_language/test_phi3v.py`
- `tests/models/decoder_only/vision_language/test_pixtral.py`
- `tests/models/multimodal/processing/test_common.py`
- `tests/models/multimodal/processing/test_llama4.py`
- `tests/models/registry.py`
- `tests/models/test_initialization.py`
- `vllm/config.py`
- `vllm/entrypoints/chat_utils.py`
- `vllm/model_executor/layers/fused_moe/configs/E=16,N=1024,device_name=AMD_Instinct_MI300X.json`
- `vllm/model_executor/layers/fused_moe/cutlass_moe.py`
- `vllm/model_executor/layers/fused_moe/fused_moe.py`
- `vllm/model_executor/layers/fused_moe/layer.py`
- `vllm/model_executor/layers/layernorm.py`
- `vllm/model_executor/layers/quantization/awq_marlin.py`
- `vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py`
- `vllm/model_executor/layers/quantization/experts_int8.py`
- `vllm/model_executor/layers/quantization/fp8.py`
- `vllm/model_executor/layers/quantization/gguf.py`
- `vllm/model_executor/layers/quantization/gptq_marlin.py`
- `vllm/model_executor/layers/quantization/moe_wna16.py`
- `vllm/model_executor/layers/quantization/quark/quark_moe.py`
- `vllm/model_executor/layers/rotary_embedding.py`
- `vllm/model_executor/model_loader/loader.py`
- `vllm/model_executor/models/llama.py`
- `vllm/model_executor/models/llama4.py`
- `vllm/model_executor/models/mllama4.py`
- `vllm/model_executor/models/registry.py`
- `vllm/model_executor/models/telechat2.py`
- `vllm/model_executor/models/teleflm.py`
- `vllm/v1/attention/backends/flash_attn.py`
- `vllm/v1/attention/backends/triton_attn.py`
- `vllm/v1/worker/gpu_model_runner.py`

## Evidence Bundle

`evidence/pull-bundles/vllm/gh-16113`
