Commits · f66603cb6fa8414705e55985d43672a4a4febbb7 · gaoqiong / flash-attention

30 Jun, 2022 1 commit
- Support batch size > 64K by swapping grid.x and grid.y · f66603cb
  Tri Dao authored Jun 29, 2022
  
  f66603cb
25 Jun, 2022 1 commit
- Bug fix: wrong smem_o write pointer for d=16 · eeca63a7
  Tri Dao authored Jun 25, 2022
  
  eeca63a7
12 Jun, 2022 3 commits
- Refactor Gmem code to store q, k, v pointers separately · 5d07483b
  Tri Dao authored Jun 12, 2022
  
  5d07483b
- Implement bwd for head dim 128 · d3e64409
  Tri Dao authored Jun 11, 2022
  
  d3e64409
- Implement fwd for head dim 128 · 0d854692
  Tri Dao authored Jun 05, 2022
  
  0d854692
03 Jun, 2022 2 commits
- Reduce smem usage for Q and dO in the backward pass · b17c6fe2
  Tri Dao authored Jun 03, 2022
```
From 4KB per buffer to 2KB per buffer. This saves us 8KB of smem (each Q and dO
have 2 buffers)
```
  b17c6fe2
- Support Turing mma instructions · 2712aa4c
  Tri Dao authored Jun 02, 2022
  
  2712aa4c
02 Jun, 2022 2 commits
- Remove softmax fp16 max · 05087332
  Tri Dao authored Jun 02, 2022
  
  05087332
- Use Cutlass gemm as WarpMma · 14dc326e
  Tri Dao authored Jun 02, 2022
  
  14dc326e
26 May, 2022 1 commit
- Rename, add benchmarking script · 9dbc491a
  Tri Dao authored May 26, 2022
  
  9dbc491a