Commits · 2d8ea9a5303b7de8865279b66c8f7e8ed2a59aee · gaoqiong / flash-attention

21 Sep, 2023 1 commit
- Swap seqlen_q and ngroups when seqlen_q=1 (h/t Daniel Haziza) · 2d8ea9a5
  Tri Dao authored Sep 20, 2023
  
  2d8ea9a5
18 Sep, 2023 2 commits
- Swap seqlen_q, nheads for MQA when seqlen_q=1 for fwd (h/t Daniel H) · 3250ff3d
  Tri Dao authored Sep 18, 2023
  
  3250ff3d
- Remove template for (IsEvenMN=T, IsEvenK=F) to speed up compilation · 43617dea
  Tri Dao authored Sep 18, 2023
  
  43617dea
17 Sep, 2023 1 commit
- Set block size to 64 x 64 for kvcache to avoid nvcc segfaults · c984208d
  Tri Dao authored Sep 17, 2023
  
  c984208d
16 Sep, 2023 1 commit
- Implement rotary embedding in flash_attn_with_kvcache · ccbb14f3
  Tri Dao authored Sep 16, 2023
  
  ccbb14f3
13 Sep, 2023 1 commit
- Simplify the implementation of KVcache attn by appending KV first · 56b7fc6e
  Tri Dao authored Sep 13, 2023
  
  56b7fc6e
12 Sep, 2023 1 commit
- Remove some unused headers · bb9beb36
  Tri Dao authored Sep 12, 2023
  
  bb9beb36
11 Sep, 2023 1 commit
- Swap seqlen_q and nheads for MQA to speed it up (h/t Daniel Haziza) · ee77b931
  Tri Dao authored Sep 10, 2023
  
  ee77b931
04 Sep, 2023 4 commits
- Implement flash_attn_with_kvcache · 37c6e054
  Tri Dao authored Sep 04, 2023
  
  37c6e054
- Remove constexpr in launch template to fix CI compilation · 6a89b2f1
  Tri Dao authored Sep 03, 2023
  
  6a89b2f1
- Bump to v2.1.2 · 1dc1b6c8
  Tri Dao authored Sep 03, 2023
  
  1dc1b6c8
- Remove unused sdPsum in dot_do_o function · 5953c4f5
  Tri Dao authored Sep 03, 2023
  
  5953c4f5
03 Sep, 2023 1 commit
- Fix splitKV combine function when local LSEs are all -inf · 26d7d92f
  Tri Dao authored Sep 03, 2023
  
  26d7d92f
01 Sep, 2023 2 commits
- Remove commented out code in bwd (#512) · 37e32feb
  Sophia Wisdom authored Sep 01, 2023
```
* Remove lots of comments

* Remove unused traits
```
  37e32feb
- Remove old code in utils.h (#511) · dd8a7549
  Sophia Wisdom authored Sep 01, 2023
  
  dd8a7549
30 Aug, 2023 1 commit
- Fix typo with lse_max == -INFINITY · 31920dda
  Tri Dao authored Aug 29, 2023
  
  31920dda
29 Aug, 2023 1 commit
- Implement splitKV attention · b1fbbd83
  Tri Dao authored Aug 29, 2023
  
  b1fbbd83
28 Aug, 2023 1 commit
- Use generate_kernels.py script from Driss Guessous · 7a983df7
  Tri Dao authored Aug 28, 2023
  
  7a983df7
25 Aug, 2023 1 commit
- Change causal mask to be aligned to bottom-right instead of top-left · 9e5e8bc9
  Tri Dao authored Aug 21, 2023
  
  9e5e8bc9
24 Aug, 2023 1 commit
- Support flash attention 2 with causal masking when KV's seq length is longer... · e07aa036
  BoxiangW authored Aug 24, 2023
```
Support flash attention 2 with causal masking when KV's seq length is longer than Q's seq length. (#436)
```
  e07aa036
17 Aug, 2023 1 commit
- [FusedDense] Run black on fused_dense.py · bcfa7c97
  Tri Dao authored Aug 16, 2023
  
  bcfa7c97
16 Aug, 2023 1 commit
- Fix Bwd NaN for varlen when seqlen_q >> seqlen_k and causal · c65b5106
  Tri Dao authored Aug 16, 2023
  
  c65b5106
13 Aug, 2023 2 commits
- Prepare for Cutlass 3.2 · dbd79237
  Tri Dao authored Aug 13, 2023
  
  dbd79237
- Update to Cutlass 3.1 · 3524e13c
  Tri Dao authored Aug 13, 2023
  
  3524e13c
01 Aug, 2023 2 commits
- Fix race condition in bwd (overwriting sK) · 1c41d2b0
  Tri Dao authored Aug 01, 2023
  
  1c41d2b0
- Fix masking of bwd when seqlen is not divisible by 128 · a4f148b6
  Tri Dao authored Jul 31, 2023
  
  a4f148b6
27 Jul, 2023 1 commit

Kirthi Shankar Sivamani authored Jul 27, 2023



* Add RNG state to kernel launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Save seed and offset for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Single thread write to global mem
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1colblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1rowblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change forward c++ APIs to save RNG state for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change backward c++ APIs to set RNG state for bprop launcher
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Python side API changes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fix; only save seeds instead of full offset
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Account for 3D grid size
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

a03f6f8e

21 Jul, 2023 1 commit
- Fix using dO stride for O, which can cause memory error in bwd · 9ee0ff1d
  Tri Dao authored Jul 20, 2023
  
  9ee0ff1d
19 Jul, 2023 1 commit

Fix compile error on MSVC · 538d570c

danthe3rd authored Jul 19, 2023

See also: https://stackoverflow.com/questions/55136414/constexpr-variable-captured-inside-lambda-loses-its-constexpr-ness

538d570c

17 Jul, 2023 1 commit
- FlashAttention-2 release · 4f285b35
  Tri Dao authored Jul 17, 2023
  
  4f285b35
26 Apr, 2023 1 commit
- [Docs] Clearer error message for bwd d > 64, bump to v1.0.4 · ad113948
  Tri Dao authored Apr 26, 2023
  
  ad113948
15 Apr, 2023 1 commit
- only 1 thread writes to global mem in fprop · 45567a25
  Kirthi Shankar Sivamani authored Apr 15, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  45567a25
13 Apr, 2023 1 commit
- Handle FlashAttnQKVPackedSplitFunc by making rng_state optional in backward · 7d25a4ec
  Kirthi Shankar Sivamani authored Apr 13, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  7d25a4ec
12 Apr, 2023 1 commit
- Support CUDA graph capture · 31018c5f
  Kirthi Shankar Sivamani authored Apr 12, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  31018c5f
15 Mar, 2023 1 commit
- Support H100 · 1b18f1b7
  Tri Dao authored Mar 15, 2023
  
  1b18f1b7
25 Jan, 2023 1 commit
- [FA] Remove unused variable rng_engine_inputs · 6b4a4821
  Tri Dao authored Jan 25, 2023
  
  6b4a4821
06 Jan, 2023 1 commit
- [Compilation] Change BOOL_SWITCH to fix Windows compilation · a1f49a2b
  Tri Dao authored Jan 06, 2023
```
Follow xFormers's DISTPATCH_BOOL. Haven't tested it on Windows.
```
  a1f49a2b
06 Dec, 2022 1 commit
- Simplify BOOL_SWITCH macro to fix compiling error on gcc 7 · 8a2ece89
  Tri Dao authored Dec 06, 2022
  
  8a2ece89
26 Nov, 2022 2 commits
- Fix typo in comments · 9bc63d1e
  Tri Dao authored Nov 25, 2022
  
  9bc63d1e
- Speed up compilation by splitting into separate .cu files · d95ee1a9
  Tri Dao authored Nov 25, 2022
  
  d95ee1a9