Commits · 3fe8479df2e65951c3d11c35c9217c9ccb60f653 · gaoqiong / flash-attention

"examples/online_serving/cohere_rerank_client.py" did not exist on "35cc35014cac8abf2ccdf03f27c7c23aabbb8a98"

28 Mar, 2024 19 commits
- allow smaller page sizes in varlen api · 3fe8479d
  skrider authored Mar 26, 2024
  
  3fe8479d
- resolve page offsets absolutely not relatively · 58611cd9
  skrider authored Mar 26, 2024
  
  58611cd9
- revert hardcoded rotcossin thread layout · 10b6f3a8
  skrider authored Feb 26, 2024
  
  10b6f3a8
- reshape rotary sin/cos copy to align with paged KV copy · 166f33fd
  skrider authored Feb 26, 2024
  
  166f33fd
- all working except rotary embedding · 04aabfb7
  skrider authored Feb 13, 2024
  
  04aabfb7
- compiles for all h but 128 · 63b35c93
  skrider authored Feb 13, 2024
  
  63b35c93
- tidy flash_fwd_kernel · c29f7313
  skrider authored Feb 11, 2024
  
  c29f7313
- remove print statements · 3f2484ee
  skrider authored Feb 11, 2024
  
  3f2484ee
- allow small page sizes in flash api · e524c2ca
  skrider authored Feb 11, 2024
  
  e524c2ca
- paged copy refactor working for page size 256 · b1c18ca1
  skrider authored Feb 11, 2024
  
  b1c18ca1
- tests passing for single page k · 446204c7
  skrider authored Feb 11, 2024
  
  446204c7
- rearrange initial offset computation · a3e06cd5
  skrider authored Feb 11, 2024
  
  a3e06cd5
- implement kv page iteration functions · f67a6edf
  skrider authored Feb 11, 2024
  
  f67a6edf
- only test trivial block size · d4da6bfc
  skrider authored Feb 11, 2024
  
  d4da6bfc
- reshape gmem copy · a232f754
  skrider authored Feb 11, 2024
  
  a232f754
- add print statements for debugging · 0094cc95
  skrider authored Feb 09, 2024
  
  0094cc95
- add print statements for debugging · a4049ac8
  skrider authored Feb 08, 2024
  
  a4049ac8
- Add the option for the macro and note (#893) · 23e8fa5a
  Driss Guessous authored Mar 27, 2024
  
  23e8fa5a
- Minor fix in compute_attn_1rowblock_splitkv (#900) · 3e9414f1
  ljss authored Mar 28, 2024
  
  3e9414f1
15 Mar, 2024 2 commits
- Add in, macrosf for defining __grid_constant__ (#852) · 4a73e903
  Driss Guessous authored Mar 15, 2024
  
  4a73e903
- Enable paged attention in varlen forward (#831) · 2a15840f
  Grigory Sizov authored Mar 15, 2024
```
* Enable paged attention in varlen forward

* Format + fix padding
```
  2a15840f
14 Mar, 2024 1 commit
- Make nvcc threads configurable via environment variable (#885) · 50896ec5
  Chirag Jain authored Mar 14, 2024
  
  50896ec5
21 Feb, 2024 2 commits
- Enable headdim 256 backward on consumer GPUs (Ampere, Ada) · 2406f288
  Tri Dao authored Feb 21, 2024
  
  2406f288
- Update Cutlass to v3.4.1 · 4d6b794b
  Tri Dao authored Feb 20, 2024
  
  4d6b794b
20 Feb, 2024 1 commit
- Don't need to reduce row_sum during online softmax · b32efb1a
  Tri Dao authored Feb 20, 2024
  
  b32efb1a
10 Feb, 2024 2 commits
- Fix dv = torch::empty_like(k) for mha_bwd_varlen as well · d9a5cb29
  Tri Dao authored Feb 10, 2024
  
  d9a5cb29
- fix backward for when query and key have different contiguity (#818) · 2423cca3
  Brian Hirsh authored Feb 10, 2024
  
  2423cca3
08 Feb, 2024 1 commit
- Fix Windows build (#816) · 46879364
  Grigory Sizov authored Feb 08, 2024
  
  46879364
30 Jan, 2024 1 commit

Preprocessor switches to control functionality (#788) · 0658e320

Jeremy Reizenstein authored Jan 30, 2024



For faster and smaller builds in some simple cases,
provide switches to allow disabling
-backward
-alibi
-uneven k
-dropout
-local attention
Co-authored-by: Jeremy Francis Reizenstein <bottler@users.noreply.github.com>

0658e320

23 Jan, 2024 2 commits
- Implement page KV cache · 54e80a38
  Tri Dao authored Jan 22, 2024
```
Co-authored-by: ljss <450993438@qq.com>
```
  54e80a38
- Use int64_t instead of uint32_t in kernel_traits.h · 36bc29ed
  Tri Dao authored Jan 22, 2024
  
  36bc29ed
22 Jan, 2024 1 commit
- Use int64_t instead of uint32_t for index_t · 000b67f5
  Tri Dao authored Jan 22, 2024
  
  000b67f5
21 Jan, 2024 7 commits
- Remove configure in bwd kernel launch · ea8a25ca
  Tri Dao authored Jan 21, 2024
  
  ea8a25ca
- Add split-kv and M<->H swap to varlen forward decoding attention (#754) · af01244d
  Grigory Sizov authored Jan 22, 2024
```
* Add split-k, M<->H to varseq path

* skip M<->H when dropout>0, fix LSE
```
  af01244d
- Update cutlass to v3.4.0 · 8f4d82cf
  Tri Dao authored Jan 20, 2024
  
  8f4d82cf
- Move rotary device functions to a separate file · 395e5a0d
  Tri Dao authored Jan 20, 2024
  
  395e5a0d
- Remove unused kernel_traits file · 3e2c827d
  Tri Dao authored Jan 20, 2024
  
  3e2c827d
- Refactor masking in fwd pass into 1 object · 66a127ae
  Tri Dao authored Jan 20, 2024
  
  66a127ae
- Change inline to __forceinline__, use __grid_constant__ param · ed4959b2
  Tri Dao authored Jan 20, 2024
  
  ed4959b2
20 Jan, 2024 1 commit
- Make Softmax an object · 6f706eff
  Tri Dao authored Jan 15, 2024
  
  6f706eff