Commits · 9b0bc978729d5470aaf5667b8f65df6fa2b3a007 · gaoqiong / flash-attention

31 Oct, 2022 10 commits
- Fix race condition in Triton fwd · 9b0bc978
  Tri Dao authored Oct 31, 2022
  
  9b0bc978
- Fix EVEN_M & EVEN_HEADDIM for headdim=40 in Triton bwd · 215930bc
  Tri Dao authored Oct 31, 2022
  
  215930bc
- Add debug_barrier for all headdims in Triton bwd · 4f81aff4
  Tri Dao authored Oct 31, 2022
  
  4f81aff4
- Disable some autotune configs that give wrong results in Triton bwd · bedcbd6a
  Tri Dao authored Oct 31, 2022
  
  bedcbd6a
- [WIP] Support all head dimensions up to 128 in the Triton bwd · e78d509c
  Tri Dao authored Oct 31, 2022
```
WIP because there seems to be some race conditions for head dimensions other
than 16, 32, 64, 128.
```
  e78d509c
- Support all head dimensions up to 128 in the Triton fwd · 008951f1
  Tri Dao authored Oct 30, 2022
  
  008951f1
- Support arbitrary seqlens (both q & k) in Triton bwd · b910bf14
  Tri Dao authored Oct 30, 2022
  
  b910bf14
- Support arbitrary seqlen_k in Triton bwd · dc554693
  Tri Dao authored Oct 30, 2022
  
  dc554693
- Fix Triton fwd to support seqlen not multiples of 128 · d11341fd
  Tri Dao authored Oct 30, 2022
  
  d11341fd
- Implement FlashAttention in Triton · b0c0db81
  Tri Dao authored Oct 30, 2022
  
  b0c0db81