Commits · 4f285b354796fb17df8636485b9a04df3ebbb7dc · gaoqiong / flash-attention

17 Jul, 2023 1 commit
- FlashAttention-2 release · 4f285b35
  Tri Dao authored Jul 17, 2023
  
  4f285b35
03 Jul, 2023 1 commit
- [Doc] Change total -> total_q · e8a0b4ac
  Tri Dao authored Jul 02, 2023
  
  e8a0b4ac
13 Apr, 2023 1 commit
- Handle FlashAttnQKVPackedSplitFunc by making rng_state optional in backward · 7d25a4ec
  Kirthi Shankar Sivamani authored Apr 13, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  7d25a4ec
12 Apr, 2023 1 commit
- Support CUDA graph capture · 31018c5f
  Kirthi Shankar Sivamani authored Apr 12, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  31018c5f
31 Mar, 2023 1 commit
- Add option for deterministic execution · b6aa059b
  Kirthi Shankar Sivamani authored Mar 30, 2023
  
  b6aa059b
13 Dec, 2022 1 commit
- Fix the case when dout is not contiguous · 88c4e5db
  Tri Dao authored Dec 13, 2022
  
  88c4e5db
05 Nov, 2022 1 commit
- Parallelize CUDA bwd along seqlen_k instead of seqlen_q · 55778193
  Tri Dao authored Nov 05, 2022
```
This is faster since we only need to do atomic adds on dq, instead of atomic
adds on both dk and dv.
```
  55778193
24 Oct, 2022 1 commit
- Support all head dims that are multiples of 8, up to 128 · 46fd2a20
  Tri Dao authored Oct 24, 2022
  
  46fd2a20
23 Oct, 2022 1 commit
- Split bwd on the seqlen_q dimension · a5a8806d
  Tri Dao authored Oct 23, 2022
  
  a5a8806d
21 Oct, 2022 2 commits
- Split fwd on the seqlen_q dimension · a44f48df
  Tri Dao authored Oct 21, 2022
  
  a44f48df
- Rework dropout to decouple forward and backward · 1aa6d7d9
  Tri Dao authored Oct 18, 2022
```
They don't have to have the same block size, number of threads, etc.
```
  1aa6d7d9
14 Oct, 2022 2 commits
- Fix QKV interface to allocate output in Python · 1b9facac
  Tri Dao authored Oct 14, 2022
  
  1b9facac
- Implement attention kernel that splits the batch into two · 5badfb78
  Tri Dao authored Oct 13, 2022
  
  5badfb78
04 Jul, 2022 2 commits
- Do P * dP (pointwise) in the bwd in fp32 instead of fp16 · a5559a0e
  Tri Dao authored Jul 03, 2022
  
  a5559a0e
- Implement cross attention · 6c3a8c65
  Tri Dao authored Jun 30, 2022
  
  6c3a8c65
02 Jun, 2022 1 commit
- Rename src -> flash_attn · 5a61cb77
  Tri Dao authored Jun 01, 2022
  
  5a61cb77
29 May, 2022 1 commit
- Reorganize directories, add banner figure · 67c37795
  Tri Dao authored May 29, 2022
  
  67c37795
26 May, 2022 1 commit
- Rename, add benchmarking script · 9dbc491a
  Tri Dao authored May 26, 2022
  
  9dbc491a
20 May, 2022 1 commit
- First release · 1fcbe6f0
  Tri Dao authored May 20, 2022
  
  1fcbe6f0