Commits · 5d079fdd7a20bbd354080a80b20b5c36c93898d4 · gaoqiong / flash-attention

22 Mar, 2023 1 commit
- [Triton] Fix benchmark_causal, mention Triton version · 5d079fdd
  Tri Dao authored Mar 22, 2023
  
  5d079fdd
14 Dec, 2022 1 commit
- [Triton] Avoid einops repeat by using Tensor.expand · 6b5f271c
  Tri Dao authored Dec 14, 2022
  
  6b5f271c
22 Nov, 2022 1 commit
- [Triton] Fix variable name from qkv to kv (h/t FrankZijlstra) · b8ccd200
  Tri Dao authored Nov 22, 2022
  
  b8ccd200
07 Nov, 2022 1 commit
- Set num_warps=4 for headdim=64 in Triton fw (h/t Michael Benesty) · 908a5b22
  Tri Dao authored Nov 07, 2022
  
  908a5b22
06 Nov, 2022 1 commit
- Fix pipelining bug in Triton bwd with bias_type=matrix · 74797571
  Tri Dao authored Nov 06, 2022
  
  74797571
05 Nov, 2022 1 commit
- Parallelize CUDA bwd along seqlen_k instead of seqlen_q · 55778193
  Tri Dao authored Nov 05, 2022
```
This is faster since we only need to do atomic adds on dq, instead of atomic
adds on both dk and dv.
```
  55778193
04 Nov, 2022 3 commits
- Fix more race condition in Triton bwd when there's bias · 62025e1a
  Tri Dao authored Nov 04, 2022
  
  62025e1a
- Fix race condition in Triton bwd when there's bias · ff78ea41
  Tri Dao authored Nov 04, 2022
  
  ff78ea41
- Implement attention bias for Triton version · 86862cfd
  Tri Dao authored Nov 04, 2022
  
  86862cfd
03 Nov, 2022 1 commit
- Fix race condition for Triton bwd for headdim 48 and 96 · 470010f5
  Tri Dao authored Nov 03, 2022
  
  470010f5
02 Nov, 2022 1 commit
- Fix race condition in Triton bwd for non-po2 headdims · aacc10fb
  Tri Dao authored Nov 02, 2022
  
  aacc10fb
01 Nov, 2022 2 commits
- Avoid memcpy in the Triton bwd · 1fb12afd
  Tri Dao authored Nov 01, 2022
  
  1fb12afd
- Fix race conditions in the Triton bwd for headdim=64 · 731f154d
  Tri Dao authored Nov 01, 2022
  
  731f154d
31 Oct, 2022 10 commits
- Fix race condition in Triton fwd · 9b0bc978
  Tri Dao authored Oct 31, 2022
  
  9b0bc978
- Fix EVEN_M & EVEN_HEADDIM for headdim=40 in Triton bwd · 215930bc
  Tri Dao authored Oct 31, 2022
  
  215930bc
- Add debug_barrier for all headdims in Triton bwd · 4f81aff4
  Tri Dao authored Oct 31, 2022
  
  4f81aff4
- Disable some autotune configs that give wrong results in Triton bwd · bedcbd6a
  Tri Dao authored Oct 31, 2022
  
  bedcbd6a
- [WIP] Support all head dimensions up to 128 in the Triton bwd · e78d509c
  Tri Dao authored Oct 31, 2022
```
WIP because there seems to be some race conditions for head dimensions other
than 16, 32, 64, 128.
```
  e78d509c
- Support all head dimensions up to 128 in the Triton fwd · 008951f1
  Tri Dao authored Oct 30, 2022
  
  008951f1
- Support arbitrary seqlens (both q & k) in Triton bwd · b910bf14
  Tri Dao authored Oct 30, 2022
  
  b910bf14
- Support arbitrary seqlen_k in Triton bwd · dc554693
  Tri Dao authored Oct 30, 2022
  
  dc554693
- Fix Triton fwd to support seqlen not multiples of 128 · d11341fd
  Tri Dao authored Oct 30, 2022
  
  d11341fd
- Implement FlashAttention in Triton · b0c0db81
  Tri Dao authored Oct 30, 2022
  
  b0c0db81