Commits · ae137ed17afd71db169fff816c6ca43096299756 · gaoqiong / flash-attention

11 Dec, 2022 1 commit
- [LayerNorm] Fuse LayerScale · ae137ed1
  Tri Dao authored Dec 10, 2022
  
  ae137ed1
09 Dec, 2022 1 commit
- [LayerNorm] Support all dimensions up to 6k (if divisible by 8) · 8c6609ae
  Tri Dao authored Dec 08, 2022
  
  8c6609ae
06 Dec, 2022 1 commit
- Simplify BOOL_SWITCH macro to fix compiling error on gcc 7 · 8a2ece89
  Tri Dao authored Dec 06, 2022
  
  8a2ece89
05 Dec, 2022 1 commit
- [Docs] Mention FasterTransformer integration · a84d0728
  Tri Dao authored Dec 05, 2022
  
  a84d0728
29 Nov, 2022 2 commits
- Update configs, add results · 4a6eaa9f
  Tri Dao authored Nov 29, 2022
  
  4a6eaa9f
- Release training code · 0bf5e500
  Tri Dao authored Nov 28, 2022
  
  0bf5e500
26 Nov, 2022 2 commits
- Fix typo in comments · 9bc63d1e
  Tri Dao authored Nov 25, 2022
  
  9bc63d1e
- Speed up compilation by splitting into separate .cu files · d95ee1a9
  Tri Dao authored Nov 25, 2022
  
  d95ee1a9
25 Nov, 2022 1 commit
- [Docs] Clarify OpenFold speedup · b784ed73
  Tri Dao authored Nov 25, 2022
  
  b784ed73
23 Nov, 2022 2 commits
- [Docs] Mention OpenFold · d9021ae4
  Tri Dao authored Nov 23, 2022
  
  d9021ae4
- [ViT] Use dropout_add_ln for the 1st layer norm · 1feb9426
  Tri Dao authored Nov 23, 2022
  
  1feb9426
22 Nov, 2022 2 commits
- [Docs] Capitalize the bibtex citation · 45bcf37b
  Tri Dao authored Nov 22, 2022
  
  45bcf37b
- [Triton] Fix variable name from qkv to kv (h/t FrankZijlstra) · b8ccd200
  Tri Dao authored Nov 22, 2022
  
  b8ccd200
21 Nov, 2022 1 commit
- Bump version to 0.2.1 · 05481617
  Tri Dao authored Nov 20, 2022
  
  05481617
18 Nov, 2022 2 commits
- Add PatchEmbed · 0fa5c0d7
  Tri Dao authored Nov 17, 2022
  
  0fa5c0d7
- Add __init__.py files to subdirectories for installation · ece539ab
  Tri Dao authored Nov 17, 2022
  
  ece539ab
17 Nov, 2022 2 commits
- [LayerNorm] Compile for both sm70 and sm80 · 39ed597b
  Tri Dao authored Nov 17, 2022
  
  39ed597b
- [Rotary] Customize base, support seqlen_offset · 71f674ae
  Tri Dao authored Nov 17, 2022
  
  71f674ae
15 Nov, 2022 7 commits
- Set version to 0.2.0 (instead of 0.2) · d6ef701a
  Tri Dao authored Nov 15, 2022
  
  d6ef701a
- Update pip install instructions, bump to 0.2 · 4040256b
  Tri Dao authored Nov 15, 2022
  
  4040256b
- Merge pull request #75 from lucidrains/main · 56aa4903
  Tri Dao authored Nov 15, 2022
```
allow for uploading to pypi
```
  56aa4903
- update manifest · dcf39865
  Phil Wang authored Nov 15, 2022
  
  dcf39865
- allow for uploading to pypi · b0eac329
  Phil Wang authored Nov 15, 2022
  
  b0eac329
- Mention that some CUDA extensions have only been tested on A100s · 43ab0b52
  Tri Dao authored Nov 15, 2022
  
  43ab0b52
- [LayerNorm] Check cuda error after querying ctas_per_sm · e4d3013e
  Tri Dao authored Nov 15, 2022
  
  e4d3013e
14 Nov, 2022 9 commits
- Mention DeepSpeed inference in usage.md · b0ed0a73
  Tri Dao authored Nov 14, 2022
  
  b0ed0a73
- Mention AITemplate Stable Diffusion in usage.md · 25387b24
  Tri Dao authored Nov 14, 2022
  
  25387b24
- Add GPT and ViT models · 2e33fc8e
  Tri Dao authored Nov 13, 2022
  
  2e33fc8e
- Add MLP, MHA, Block, Embedding modules · d4b320b3
  Tri Dao authored Nov 13, 2022
  
  d4b320b3
- Add fused_dense and dropout_add_layernorm CUDA extensions · fa6d1ce4
  Tri Dao authored Nov 13, 2022
  
  fa6d1ce4
- Link to Colossal-AI's stable diffusion in usage.md · b92f2c3b
  Tri Dao authored Nov 13, 2022
  
  b92f2c3b
- Make nccl operations async in CrossEntropyLossParallel · 343492ec
  Tri Dao authored Nov 13, 2022
  
  343492ec
- Update README · 3dda4f76
  Tri Dao authored Nov 13, 2022
  
  3dda4f76
- Add a page on where FlashAttention is being used · 79160a69
  Tri Dao authored Nov 13, 2022
  
  79160a69
13 Nov, 2022 3 commits
- Skip flash_attn_split test · a8fec99a
  Tri Dao authored Nov 13, 2022
  
  a8fec99a
- Don't enforce bitwise consistency for dq in race condition test · 9d3116ad
  Tri Dao authored Nov 13, 2022
```
Since we could be parallelizing over seqlen_k
```
  9d3116ad
- Add fused cross entropy loss · 7c995381
  Tri Dao authored Nov 12, 2022
  
  7c995381
10 Nov, 2022 1 commit
- Remove RotaryEmbedding from FlashAttention module · 55797f32
  Tri Dao authored Nov 10, 2022
```
To avoid import error if one doesn't have rotary_emb installed
```
  55797f32
09 Nov, 2022 1 commit
- Fix out-of-bound memory read · 6998e0ec
  Tri Dao authored Nov 09, 2022
  
  6998e0ec
07 Nov, 2022 1 commit
- Set num_warps=4 for headdim=64 in Triton fw (h/t Michael Benesty) · 908a5b22
  Tri Dao authored Nov 07, 2022
  
  908a5b22