Commits · a5559a0e752ec9f8ce7157e1d4db9f08593638c3 · gaoqiong / flash-attention

04 Jul, 2022 2 commits
- Do P * dP (pointwise) in the bwd in fp32 instead of fp16 · a5559a0e
  Tri Dao authored Jul 03, 2022
  
  a5559a0e
- Implement cross attention · 6c3a8c65
  Tri Dao authored Jun 30, 2022
  
  6c3a8c65
03 Jul, 2022 3 commits
- Merge pull request #18 from gahdritz/main · 01947bc9
  Tri Dao authored Jul 02, 2022
```
Slightly improve installation process
```
  01947bc9
- Add missing __init__.py · af4a9ce0
  Gustaf authored Jul 03, 2022
  
  af4a9ce0
- Add einops installation to setup.py · 440e9c49
  Gustaf authored Jul 03, 2022
  
  440e9c49
30 Jun, 2022 1 commit
- Support batch size > 64K by swapping grid.x and grid.y · f66603cb
  Tri Dao authored Jun 29, 2022
  
  f66603cb
27 Jun, 2022 1 commit
- Add README section on issues · 450b64fe
  Tri Dao authored Jun 27, 2022
  
  450b64fe
26 Jun, 2022 2 commits
- Add type check (fp16) in the forward pass · c0daa62e
  Tri Dao authored Jun 26, 2022
  
  c0daa62e
- Fix race condition in backward pass (smem_dq) · ea38d3d2
  Tri Dao authored Jun 25, 2022
  
  ea38d3d2
25 Jun, 2022 1 commit
- Bug fix: wrong smem_o write pointer for d=16 · eeca63a7
  Tri Dao authored Jun 25, 2022
  
  eeca63a7
14 Jun, 2022 2 commits
- More explanation · 765741c1
  Dan Fu authored Jun 14, 2022
  
  765741c1
- Speedup graph for A100, d128 · 2d5b2483
  Dan Fu authored Jun 14, 2022
  
  2d5b2483
12 Jun, 2022 3 commits
- Refactor Gmem code to store q, k, v pointers separately · 5d07483b
  Tri Dao authored Jun 12, 2022
  
  5d07483b
- Implement bwd for head dim 128 · d3e64409
  Tri Dao authored Jun 11, 2022
  
  d3e64409
- Implement fwd for head dim 128 · 0d854692
  Tri Dao authored Jun 05, 2022
  
  0d854692
05 Jun, 2022 2 commits
- Broken link · 0a398dfc
  Dan Fu authored Jun 04, 2022
  
  0a398dfc
- T4 · bd60750e
  Dan Fu authored Jun 04, 2022
  
  bd60750e
04 Jun, 2022 3 commits
- Set block size of SM75 fwd to 256 if there's no dropout · 321c57d0
  Tri Dao authored Jun 04, 2022
```
This speeds up the fwd by 1.5x.
```
  321c57d0
- Edit README: support Turing (SM75) · f2d8d410
  Tri Dao authored Jun 04, 2022
  
  f2d8d410
- Don't use Smem_dp_sum in backward pass · d380e87f
  Tri Dao authored Jun 04, 2022
```
To reduce smem usage for SM75
```
  d380e87f
03 Jun, 2022 2 commits
- Reduce smem usage for Q and dO in the backward pass · b17c6fe2
  Tri Dao authored Jun 03, 2022
```
From 4KB per buffer to 2KB per buffer. This saves us 8KB of smem (each Q and dO
have 2 buffers)
```
  b17c6fe2
- Support Turing mma instructions · 2712aa4c
  Tri Dao authored Jun 02, 2022
  
  2712aa4c
02 Jun, 2022 7 commits
- Remove softmax fp16 max · 05087332
  Tri Dao authored Jun 02, 2022
  
  05087332
- Use Cutlass gemm as WarpMma · 14dc326e
  Tri Dao authored Jun 02, 2022
  
  14dc326e
- Remove old backward · e78e7c95
  Tri Dao authored Jun 02, 2022
  
  e78e7c95
- Add Cutlass as submodule · 512c98ee
  Tri Dao authored Jun 02, 2022
  
  512c98ee
- 3090 speedup · ad6c694b
  Dan Fu authored Jun 01, 2022
  
  ad6c694b
- Rename src -> flash_attn · 5a61cb77
  Tri Dao authored Jun 01, 2022
  
  5a61cb77
- Support SM86 GPUs · c41479d6
  Tri Dao authored Jun 01, 2022
  
  c41479d6
30 May, 2022 4 commits
- Citation · 4b7cfb5f
  Dan Fu authored May 30, 2022
  
  4b7cfb5f
- Jpg resolution · 963173fc
  Dan Fu authored May 30, 2022
  
  963173fc
- Fix jpg · cd04d298
  Dan Fu authored May 30, 2022
  
  cd04d298
- Add paper arXiv link · a7874518
  Tri Dao authored May 29, 2022
  
  a7874518
29 May, 2022 3 commits
- Edit roadmap · d9fff84b
  Tri Dao authored May 29, 2022
  
  d9fff84b
- Convert banner figure from pdf to jpg · e4ffe5d5
  Tri Dao authored May 29, 2022
  
  e4ffe5d5
- Reorganize directories, add banner figure · 67c37795
  Tri Dao authored May 29, 2022
  
  67c37795
28 May, 2022 1 commit
- Make png images into jpg for dark mode · 7025a092
  Dan Fu authored May 28, 2022
  
  7025a092
27 May, 2022 2 commits
- README typo · 4decc3c1
  Dan Fu authored May 27, 2022
  
  4decc3c1
- Add speedup to README · dc6d1300
  Dan Fu authored May 27, 2022
```
Update images

Update images

Update description
```
  dc6d1300
26 May, 2022 1 commit
- Rename, add benchmarking script · 9dbc491a
  Tri Dao authored May 26, 2022
  
  9dbc491a