Commits · ce26d3d73d07e9779c5ba6fb2ca3bc187a34c6cc · gaoqiong / flash-attention

07 Jan, 2023 1 commit
- [LayerNorm] Implement RMS Norm · 6738d947
  Tri Dao authored Jan 06, 2023
  
  6738d947
04 Jan, 2023 1 commit
- [Gen] Add option to run generation with FT attention kernel · a668890f
  Tri Dao authored Jan 03, 2023
  
  a668890f
02 Jan, 2023 2 commits
- [TP] Put parallel embeddings in separate modules · 4cab4de5
  Tri Dao authored Jan 02, 2023
  
  4cab4de5
- [FusedDense] Limit matrix dims to 2M (instead of 64k) · 1ec09ebd
  Tri Dao authored Jan 01, 2023
  
  1ec09ebd
01 Jan, 2023 3 commits
- [Bert] Fix embedding layer norm before embedding dropout · 714c1b4f
  Tri Dao authored Jan 01, 2023
  
  714c1b4f
- [GPT] Refactor function to shard state_dict for TensorParallel · ef1ba918
  Tri Dao authored Jan 01, 2023
  
  ef1ba918
- [FusedDense] Kick off input all_gather before weight dtype conversion · 65b4064b
  Tri Dao authored Dec 31, 2022
  
  65b4064b
30 Dec, 2022 1 commit
- [Docs] Mention that XPos's scale_base is recommended to be 512 · 85b8e3d3
  Tri Dao authored Dec 29, 2022
  
  85b8e3d3
28 Dec, 2022 2 commits
- Bump to v0.2.6 · a6ec1782
  Tri Dao authored Dec 27, 2022
  
  a6ec1782
- Implement generation for GPT · 63670fd8
  Tri Dao authored Dec 27, 2022
  
  63670fd8
27 Dec, 2022 3 commits
- Support loading GPT2 weights from Huggingface · 9d797d88
  Tri Dao authored Dec 27, 2022
  
  9d797d88
- Tweak CrossEntropyLoss to take process_group in init · c6ecd40a
  Tri Dao authored Dec 27, 2022
  
  c6ecd40a
- Implement Tensor Parallel for GPT model · b4018a50
  Tri Dao authored Dec 25, 2022
  
  b4018a50
25 Dec, 2022 3 commits
- Implement Tensor Parallel for GPT2Embeddings · 78225c53
  Tri Dao authored Dec 25, 2022
  
  78225c53
- Implement Tensor Parallel for transformer Block · a8cfe515
  Tri Dao authored Dec 25, 2022
  
  a8cfe515
- Implement TensorParallel for MHA · 1e712ea8
  Tri Dao authored Dec 24, 2022
  
  1e712ea8
24 Dec, 2022 1 commit
- Implement TensorParallel for FusedDense and FusedDenseGeluDense · 226a1b72
  Tri Dao authored Dec 23, 2022
  
  226a1b72
23 Dec, 2022 2 commits
- Add smoothing for CrossEntropyParallel, rename to CrossEntropyLoss · dff68c2b
  Tri Dao authored Dec 23, 2022
  
  dff68c2b
- Simplify FusedDense · e68ebbe8
  Tri Dao authored Dec 22, 2022
  
  e68ebbe8
21 Dec, 2022 1 commit
- Implement XPos (Sun et al.) · 496e4f52
  Tri Dao authored Dec 21, 2022
  
  496e4f52
20 Dec, 2022 1 commit
- Implement last_layer_subset optimization for BERT · 13cdceb3
  Tri Dao authored Dec 19, 2022
  
  13cdceb3
19 Dec, 2022 1 commit
- Implement BERT · 5fb6df0e
  Tri Dao authored Dec 18, 2022
  
  5fb6df0e
17 Dec, 2022 1 commit
- add asserts for sin shape · ee8984d2
  Alexander Ploshkin authored Dec 17, 2022
  
  ee8984d2
16 Dec, 2022 1 commit
- fix slicing dimensions · c7c66976
  Alexander Ploshkin authored Dec 16, 2022
  
  c7c66976
15 Dec, 2022 1 commit
- Remove redundant shape asserts in rotary embeddings · 96656b93
  Alexander Ploshkin authored Dec 15, 2022
  
  96656b93
14 Dec, 2022 1 commit
- [Triton] Avoid einops repeat by using Tensor.expand · 6b5f271c
  Tri Dao authored Dec 14, 2022
  
  6b5f271c
13 Dec, 2022 2 commits
- Fix the case when dout is not contiguous · 88c4e5db
  Tri Dao authored Dec 13, 2022
  
  88c4e5db
- [LayerNorm] Support taking subset of input or subset of output · 5db33051
  Tri Dao authored Dec 12, 2022
  
  5db33051
11 Dec, 2022 1 commit
- [LayerNorm] Fuse LayerScale · ae137ed1
  Tri Dao authored Dec 10, 2022
  
  ae137ed1
09 Dec, 2022 1 commit
- [LayerNorm] Support all dimensions up to 6k (if divisible by 8) · 8c6609ae
  Tri Dao authored Dec 08, 2022
  
  8c6609ae
23 Nov, 2022 1 commit
- [ViT] Use dropout_add_ln for the 1st layer norm · 1feb9426
  Tri Dao authored Nov 23, 2022
  
  1feb9426
22 Nov, 2022 1 commit
- [Triton] Fix variable name from qkv to kv (h/t FrankZijlstra) · b8ccd200
  Tri Dao authored Nov 22, 2022
  
  b8ccd200
21 Nov, 2022 1 commit
- Bump version to 0.2.1 · 05481617
  Tri Dao authored Nov 20, 2022
  
  05481617
18 Nov, 2022 2 commits
- Add PatchEmbed · 0fa5c0d7
  Tri Dao authored Nov 17, 2022
  
  0fa5c0d7
- Add __init__.py files to subdirectories for installation · ece539ab
  Tri Dao authored Nov 17, 2022
  
  ece539ab
17 Nov, 2022 1 commit
- [Rotary] Customize base, support seqlen_offset · 71f674ae
  Tri Dao authored Nov 17, 2022
  
  71f674ae
14 Nov, 2022 4 commits
- Add GPT and ViT models · 2e33fc8e
  Tri Dao authored Nov 13, 2022
  
  2e33fc8e
- Add MLP, MHA, Block, Embedding modules · d4b320b3
  Tri Dao authored Nov 13, 2022
  
  d4b320b3
- Add fused_dense and dropout_add_layernorm CUDA extensions · fa6d1ce4
  Tri Dao authored Nov 13, 2022
  
  fa6d1ce4
- Make nccl operations async in CrossEntropyLossParallel · 343492ec
  Tri Dao authored Nov 13, 2022
  
  343492ec