---
id: pr-flash-attention-1236
repo: Dao-AILab/flash-attention
pr: 1236
title: FA3 kvcache + split kv + gqa parallelization
author: jayhshah
date: '2024-10-15'
url: https://github.com/Dao-AILab/flash-attention/pull/1236
source_category: upstream-code
architectures:
- sm90
tags:
- FA3
- arch_pipeline
- attention_kv
- benchmark_test
- fp8
- scheduler_autotune
techniques:
- async-copy
- autotuning
- cache-policy
- epilogue-fusion
- pipeline-stages
- tile-scheduling
hardware_features:
- fp8
- hopper
- tma
kernel_types:
- attention
- runtime
- scheduler
languages:
- cuda-cpp
captured_at: '2026-05-16'
status: merged
merge_sha: a5a75274bc5dc3df59d8a8e28a0a841a438002ad
inclusion_reason: Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.
changed_paths:
- hopper/benchmark_flash_attention_fp8.py
- hopper/benchmark_split_kv.py
- hopper/block_info.h
- hopper/combine.h
- hopper/epilogue_fwd_sm90_tma.hpp
- hopper/flash.h
- hopper/flash_api.cpp
- hopper/flash_attn_interface.py
- hopper/flash_fwd_hdim128_bf16_gqa16_sm90.cu
- hopper/flash_fwd_hdim128_bf16_gqa2_sm90.cu
- hopper/flash_fwd_hdim128_bf16_gqa32_sm90.cu
- hopper/flash_fwd_hdim128_bf16_gqa4_sm90.cu
- hopper/flash_fwd_hdim128_bf16_gqa8_sm90.cu
- hopper/flash_fwd_hdim128_e4m3_gqa16_sm90.cu
- hopper/flash_fwd_hdim128_e4m3_gqa2_sm90.cu
- hopper/flash_fwd_hdim128_e4m3_gqa32_sm90.cu
artifact_dir: evidence/pull-bundles/flash-attention/gh-1236
head_ref_oid: 9b6cba16c1e1b705792e9c4ba57f2e87547223ab
---
# FA3 kvcache + split kv + gqa parallelization

- Repository: `Dao-AILab/flash-attention`
- PR: [1236](https://github.com/Dao-AILab/flash-attention/pull/1236)
- Decision: `Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.`
- Reason: Preserve page/KV/layout and plan-run contracts; profile memory traffic separately from math utilization.

## Changed Paths

- `hopper/benchmark_flash_attention_fp8.py`
- `hopper/benchmark_split_kv.py`
- `hopper/block_info.h`
- `hopper/combine.h`
- `hopper/epilogue_fwd_sm90_tma.hpp`
- `hopper/flash.h`
- `hopper/flash_api.cpp`
- `hopper/flash_attn_interface.py`
- `hopper/flash_fwd_hdim128_bf16_gqa16_sm90.cu`
- `hopper/flash_fwd_hdim128_bf16_gqa2_sm90.cu`
- `hopper/flash_fwd_hdim128_bf16_gqa32_sm90.cu`
- `hopper/flash_fwd_hdim128_bf16_gqa4_sm90.cu`
- `hopper/flash_fwd_hdim128_bf16_gqa8_sm90.cu`
- `hopper/flash_fwd_hdim128_e4m3_gqa16_sm90.cu`
- `hopper/flash_fwd_hdim128_e4m3_gqa2_sm90.cu`
- `hopper/flash_fwd_hdim128_e4m3_gqa32_sm90.cu`

## Evidence Bundle

`evidence/pull-bundles/flash-attention/gh-1236`
