Commits · 1aa6d7d9b60bf8fbb5584f057934bdee15ed33fe · gaoqiong / flash-attention

21 Oct, 2022 1 commit
- Rework dropout to decouple forward and backward · 1aa6d7d9
  Tri Dao authored Oct 18, 2022
```
They don't have to have the same block size, number of threads, etc.
```
  1aa6d7d9
05 Oct, 2022 1 commit
- Replace BOOL_SWITCH with FP16_SWITCH to work around MSVC bug with constexpr variables and templates · 9d7fd5b6
  Eric Engelhart authored Oct 03, 2022
  
  9d7fd5b6
12 Sep, 2022 1 commit
- Use block_size=128 for d=128 on SM86 to avoid exceeding smem limit · 8166063a
  Tri Dao authored Sep 12, 2022
  
  8166063a
11 Jul, 2022 1 commit
- Don't nest BOOL_SWITCH to work around gcc 7 bug · bc2c2102
  Tri Dao authored Jul 11, 2022
  
  bc2c2102
10 Jul, 2022 1 commit
- Implement for bf16 · de19de7a
  Tri Dao authored Jul 09, 2022
  
  de19de7a
04 Jul, 2022 1 commit
- Implement cross attention · 6c3a8c65
  Tri Dao authored Jun 30, 2022
  
  6c3a8c65
30 Jun, 2022 1 commit
- Support batch size > 64K by swapping grid.x and grid.y · f66603cb
  Tri Dao authored Jun 29, 2022
  
  f66603cb
12 Jun, 2022 2 commits
- Implement bwd for head dim 128 · d3e64409
  Tri Dao authored Jun 11, 2022
  
  d3e64409
- Implement fwd for head dim 128 · 0d854692
  Tri Dao authored Jun 05, 2022
  
  0d854692
04 Jun, 2022 2 commits
- Set block size of SM75 fwd to 256 if there's no dropout · 321c57d0
  Tri Dao authored Jun 04, 2022
```
This speeds up the fwd by 1.5x.
```
  321c57d0
- Don't use Smem_dp_sum in backward pass · d380e87f
  Tri Dao authored Jun 04, 2022
```
To reduce smem usage for SM75
```
  d380e87f
03 Jun, 2022 1 commit
- Support Turing mma instructions · 2712aa4c
  Tri Dao authored Jun 02, 2022
  
  2712aa4c
26 May, 2022 1 commit
- Rename, add benchmarking script · 9dbc491a
  Tri Dao authored May 26, 2022
  
  9dbc491a
20 May, 2022 1 commit
- First release · 1fcbe6f0
  Tri Dao authored May 20, 2022
  
  1fcbe6f0