Commits · d95ee1a95da2b138073b5eef0f09ce4de615ba5e · gaoqiong / flash-attention

26 Nov, 2022 1 commit
- Speed up compilation by splitting into separate .cu files · d95ee1a9
  Tri Dao authored Nov 25, 2022
  
  d95ee1a9
23 Oct, 2022 1 commit
- Split bwd on the seqlen_q dimension · a5a8806d
  Tri Dao authored Oct 23, 2022
  
  a5a8806d
22 Oct, 2022 1 commit
- Don't need to run configure for the forward pass · 871db479
  Tri Dao authored Oct 21, 2022
  
  871db479
21 Oct, 2022 1 commit
- Split fwd on the seqlen_q dimension · a44f48df
  Tri Dao authored Oct 21, 2022
  
  a44f48df
14 Oct, 2022 1 commit
- Implement attention kernel that splits the batch into two · 5badfb78
  Tri Dao authored Oct 13, 2022
  
  5badfb78
10 Jul, 2022 1 commit
- Implement for bf16 · de19de7a
  Tri Dao authored Jul 09, 2022
  
  de19de7a
04 Jul, 2022 3 commits
- Apply dropout scaling to dQ and dK instead of to V (in bwd) · 5b838a8b
  Tri Dao authored Jun 29, 2022
```
Theoretically this might have lower numerical error since the scaling is in
fp32 instead of fp16 (not sure, I haven't thought too carefully about it).
However, in practice, the numerical errors seem about the same.
```
  5b838a8b
- Do P * dP (pointwise) in the bwd in fp32 instead of fp16 · a5559a0e
  Tri Dao authored Jul 03, 2022
  
  a5559a0e
- Implement cross attention · 6c3a8c65
  Tri Dao authored Jun 30, 2022
  
  6c3a8c65
12 Jun, 2022 1 commit
- Refactor Gmem code to store q, k, v pointers separately · 5d07483b
  Tri Dao authored Jun 12, 2022
  
  5d07483b
26 May, 2022 1 commit
- Rename, add benchmarking script · 9dbc491a
  Tri Dao authored May 26, 2022
  
  9dbc491a
20 May, 2022 1 commit
- First release · 1fcbe6f0
  Tri Dao authored May 20, 2022
  
  1fcbe6f0