Commits · 393882bc089aff64863c260a8633b8390c90aa78 · gaoqiong / flash-attention

31 Mar, 2023 1 commit
- [LayerNorm] Implement LN with parallel residual, support dim 8k · 393882bc
  Tri Dao authored Mar 29, 2023
  
  393882bc
29 Mar, 2023 1 commit
- Implement GPT-NeoX · 993d1244
  Tri Dao authored Mar 29, 2023
  
  993d1244
22 Mar, 2023 1 commit
- Implement GPT-J · 4d87e4d8
  Tri Dao authored Mar 22, 2023
  
  4d87e4d8
23 Jan, 2023 1 commit
- [OPT] Load fp16 weights on CPU before moving to GPU · 78b7a1dc
  Tri Dao authored Jan 22, 2023
  
  78b7a1dc
19 Jan, 2023 1 commit
- [LayerNorm] Rename x1 -> residual · eb33e587
  Tri Dao authored Jan 19, 2023
  
  eb33e587
18 Jan, 2023 1 commit
- [FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP · 88173a1a
  Tri Dao authored Jan 17, 2023
  
  88173a1a
16 Jan, 2023 3 commits
- [ViT] Support timm checkpoint, add tests · 780e8eea
  Tri Dao authored Jan 16, 2023
  
  780e8eea
- [ViT] Fix extra norm_0, use new LN order in Block · ef085cfc
  Tri Dao authored Jan 15, 2023
  
  ef085cfc
- Reorder LN in Block, support OPT · ff34123b
  Tri Dao authored Jan 15, 2023
  
  ff34123b
15 Jan, 2023 1 commit
- [Gen] Make generation work with Tensor Parallel · 7c219154
  Tri Dao authored Jan 15, 2023
  
  7c219154
07 Jan, 2023 2 commits
- [Gen] Test generation with rotary embedding · 11be742a
  Tri Dao authored Jan 07, 2023
  
  11be742a
- [TP] Implement TensorParallel without sequence parallel · 93383bd5
  Tri Dao authored Jan 07, 2023
  
  93383bd5
01 Jan, 2023 2 commits
- [Bert] Fix embedding layer norm before embedding dropout · 714c1b4f
  Tri Dao authored Jan 01, 2023
  
  714c1b4f
- [GPT] Refactor function to shard state_dict for TensorParallel · ef1ba918
  Tri Dao authored Jan 01, 2023
  
  ef1ba918
28 Dec, 2022 1 commit
- Implement generation for GPT · 63670fd8
  Tri Dao authored Dec 27, 2022
  
  63670fd8
27 Dec, 2022 3 commits
- Support loading GPT2 weights from Huggingface · 9d797d88
  Tri Dao authored Dec 27, 2022
  
  9d797d88
- Tweak CrossEntropyLoss to take process_group in init · c6ecd40a
  Tri Dao authored Dec 27, 2022
  
  c6ecd40a
- Implement Tensor Parallel for GPT model · b4018a50
  Tri Dao authored Dec 25, 2022
  
  b4018a50
23 Dec, 2022 2 commits
- Add smoothing for CrossEntropyParallel, rename to CrossEntropyLoss · dff68c2b
  Tri Dao authored Dec 23, 2022
  
  dff68c2b
- Simplify FusedDense · e68ebbe8
  Tri Dao authored Dec 22, 2022
  
  e68ebbe8
21 Dec, 2022 1 commit
- Implement XPos (Sun et al.) · 496e4f52
  Tri Dao authored Dec 21, 2022
  
  496e4f52
20 Dec, 2022 1 commit
- Implement last_layer_subset optimization for BERT · 13cdceb3
  Tri Dao authored Dec 19, 2022
  
  13cdceb3
19 Dec, 2022 1 commit
- Implement BERT · 5fb6df0e
  Tri Dao authored Dec 18, 2022
  
  5fb6df0e
23 Nov, 2022 1 commit
- [ViT] Use dropout_add_ln for the 1st layer norm · 1feb9426
  Tri Dao authored Nov 23, 2022
  
  1feb9426
18 Nov, 2022 1 commit
- Add __init__.py files to subdirectories for installation · ece539ab
  Tri Dao authored Nov 17, 2022
  
  ece539ab
14 Nov, 2022 1 commit
- Add GPT and ViT models · 2e33fc8e
  Tri Dao authored Nov 13, 2022
  
  2e33fc8e