Commits · e0fbaa7016e30dff62992706f39cab4a3dade7c4 · gaoqiong / flash-attention

20 Sep, 2023 3 commits
- [Gen] Simplify decode_speculative · e0fbaa70
  Tri Dao authored Sep 18, 2023
  
  e0fbaa70
- [Gen] Rename max_sequence_len->max_seqlen, sequence_len_offset->seqlen_offset · e6a80264
  Tri Dao authored Sep 18, 2023
  
  e6a80264
- Fix Llama GQA/MQA (#546) · 42832575
  Kevin Hu authored Sep 19, 2023
```
* Fix llama MQA

* Fix permute shape

* Update llama.py
```
  42832575
18 Sep, 2023 5 commits
- [Gen] Don't use ft_attention, use flash_attn_with_kvcache instead · dfe29f5e
  Tri Dao authored Sep 18, 2023
  
  dfe29f5e
- Swap seqlen_q, nheads for MQA when seqlen_q=1 for fwd (h/t Daniel H) · 3250ff3d
  Tri Dao authored Sep 18, 2023
  
  3250ff3d
- Remove template for (IsEvenMN=T, IsEvenK=F) to speed up compilation · 43617dea
  Tri Dao authored Sep 18, 2023
  
  43617dea
- [Minor] add nvcc note on bare_metal_version `RuntimeError` (#552) · fa3ddcba
  Federico Berto authored Sep 19, 2023
```
* Add nvcc note on bare_metal_version `RuntimeError`

* Run Black formatting
```
  fa3ddcba
- Don't compile for Pytorch 2.1 on CUDA 12.1 due to nvcc segfaults · 799f56fa
  Tri Dao authored Sep 17, 2023
  
  799f56fa
17 Sep, 2023 1 commit
- Set block size to 64 x 64 for kvcache to avoid nvcc segfaults · c984208d
  Tri Dao authored Sep 17, 2023
  
  c984208d
16 Sep, 2023 3 commits
- Bump to v2.2.3 · 8c8b4d36
  Tri Dao authored Sep 16, 2023
  
  8c8b4d36
- Implement rotary embedding in flash_attn_with_kvcache · ccbb14f3
  Tri Dao authored Sep 16, 2023
  
  ccbb14f3
- [CE] Implement CrossEntropyLoss in Triton · 5400fdc4
  Tri Dao authored Sep 15, 2023
  
  5400fdc4
13 Sep, 2023 2 commits
- Simplify the implementation of KVcache attn by appending KV first · 56b7fc6e
  Tri Dao authored Sep 13, 2023
  
  56b7fc6e
- Add tests for Pythia, GPT-JT, and RedPajama models · d0032700
  Tri Dao authored Sep 13, 2023
  
  d0032700
12 Sep, 2023 1 commit
- Remove some unused headers · bb9beb36
  Tri Dao authored Sep 12, 2023
  
  bb9beb36
11 Sep, 2023 4 commits
- Bump to v2.2.2 · 08c295c0
  Tri Dao authored Sep 10, 2023
  
  08c295c0
- Swap seqlen_q and nheads for MQA to speed it up (h/t Daniel Haziza) · ee77b931
  Tri Dao authored Sep 10, 2023
  
  ee77b931
- Add BigCode converters (#532) · 07005806
  Kevin Hu authored Sep 10, 2023
  
  07005806
- [Gen] Fix calling update_graph_cache in tests · 8a733cbd
  Tri Dao authored Sep 10, 2023
  
  8a733cbd
09 Sep, 2023 1 commit
- Inverse state dict for BERT (#527) · 4c91621a
  Kevin Hu authored Sep 09, 2023
  
  4c91621a
07 Sep, 2023 1 commit
- [Gen] Use flash_attn_with_kvcache in generation · a86442f0
  Tri Dao authored Sep 07, 2023
  
  a86442f0
06 Sep, 2023 2 commits
- Bump to v2.2.1 · a1576ad1
  Tri Dao authored Sep 06, 2023
  
  a1576ad1
- [Rotary] Set device before launching Triton kernel to avoid error · 97951590
  Tri Dao authored Sep 05, 2023
  
  97951590
05 Sep, 2023 4 commits
- Bump to v2.2.0 · 6d673cd9
  Tri Dao authored Sep 05, 2023
  
  6d673cd9
- Create __init__.py for ops/triton dir (#516) · 8e893f09
  Kyeongpil Kang authored Sep 06, 2023
  
  8e893f09
- Support cache_seqlens being integer · fd20f16a
  Tri Dao authored Sep 05, 2023
  
  fd20f16a
- [Gen] Refactor decoding function · 913922ca
  Tri Dao authored Sep 04, 2023
  
  913922ca
04 Sep, 2023 11 commits
- [MLP] Implement SwiGLU with torch jiterator · 3557e0bb
  Tri Dao authored Sep 04, 2023
  
  3557e0bb
- Implement flash_attn_with_kvcache · 37c6e054
  Tri Dao authored Sep 04, 2023
  
  37c6e054
- Set single threaded compilation for CUDA 12.2 so CI doesn't OOM · 4976650f
  Tri Dao authored Sep 03, 2023
  
  4976650f
- Remove constexpr in launch template to fix CI compilation · 6a89b2f1
  Tri Dao authored Sep 03, 2023
  
  6a89b2f1
- Try switching back to Cutlass 3.2.0 · 97ba7a62
  Tri Dao authored Sep 03, 2023
  
  97ba7a62
- Bump to v2.1.2 · 1dc1b6c8
  Tri Dao authored Sep 03, 2023
  
  1dc1b6c8
- Require CUDA 11.6+, clean up setup.py · 0c04943f
  Tri Dao authored Sep 03, 2023
  
  0c04943f
- Fix test_baichuan · 798858f9
  Tri Dao authored Sep 03, 2023
  
  798858f9
- [Gen] Add back num_last_tokens in gpt.py · 7b33743a
  Tri Dao authored Sep 03, 2023
  
  7b33743a
- Remove unused sdPsum in dot_do_o function · 5953c4f5
  Tri Dao authored Sep 03, 2023
  
  5953c4f5
- [Rotary] Implement varlen rotary · b28ec236
  Tri Dao authored Sep 03, 2023
  
  b28ec236
03 Sep, 2023 2 commits
- [Rotary] Clean up rotary Triton implementation a bit · 861c8257
  Tri Dao authored Sep 03, 2023
  
  861c8257
- [Rotary] Speed up rotary kernel when interleaved=True · 1c523c1c
  Tri Dao authored Sep 03, 2023
  
  1c523c1c