Commits · bdcae547c775f95c4aa890f7d69c49f3c5e51983 · gaoqiong / flash-attention

23 Jan, 2024 1 commit
- [LayerNorm] Don't exit early in the backward pass (fix #781) · bdcae547
  Tri Dao authored Jan 22, 2024
  
  bdcae547
21 Jan, 2024 1 commit
- return z_loss (#768) · d8aacc51
  Curtis "Fjord" Hawthorne authored Jan 21, 2024
  
  d8aacc51
10 Jan, 2024 1 commit
- [LayerNorm] Initialize mean and rstd tensor using x.device · c9861a03
  Tri Dao authored Jan 09, 2024
  
  c9861a03
05 Jan, 2024 2 commits
- [LayerNorm] Rename layernorm.py -> layer_norm.py · f5b308e2
  Tri Dao authored Jan 05, 2024
  
  f5b308e2
- [LayerNorm] Implement parallel layer norm in Triton · 665b55e2
  Tri Dao authored Jan 04, 2024
  
  665b55e2
04 Jan, 2024 1 commit
- [LayerNorm] Implement rowscale in Triton layernorm · aa5c6438
  Tri Dao authored Jan 04, 2024
  
  aa5c6438
20 Dec, 2023 1 commit
- [LayerNorm] Implement dropout in fused residual + LN/RMSNorm · cd089597
  Tri Dao authored Dec 19, 2023
  
  cd089597
17 Dec, 2023 1 commit
- [CrossEntropy] Implement logit_scale option · 08124c8f
  Tri Dao authored Dec 16, 2023
  
  08124c8f
01 Dec, 2023 1 commit
- [LayerNorm] Implement layer_norm_linear · 9356a1c0
  Tri Dao authored Nov 30, 2023
  
  9356a1c0
20 Nov, 2023 2 commits
- [CrossEntropy] Simplify the case of large vocab with Tensor Parallel · aaa14741
  Tri Dao authored Nov 19, 2023
  
  aaa14741
- fix flash ce mp large vocab (#673) · abf04a56
  Shijie authored Nov 20, 2023
  
  abf04a56
14 Nov, 2023 1 commit
- [LayerNorm] Add postnorm residual + LayerNorm/RMSNorm in Triton · 01771645
  Tri Dao authored Nov 13, 2023
  
  01771645
13 Nov, 2023 1 commit
- [LayerNorm] Implement residual + LayerNorm/RMSNorm in Triton · 79bd1a2d
  Tri Dao authored Nov 13, 2023
  
  79bd1a2d
24 Oct, 2023 1 commit
- [CrossEntropy] Fix triton cross_entropy_loss IMA for >=2B elements · c79de85f
  Tri Dao authored Oct 24, 2023
  
  c79de85f
16 Sep, 2023 1 commit
- [CE] Implement CrossEntropyLoss in Triton · 5400fdc4
  Tri Dao authored Sep 15, 2023
  
  5400fdc4
11 Sep, 2023 1 commit
- [Gen] Fix calling update_graph_cache in tests · 8a733cbd
  Tri Dao authored Sep 10, 2023
  
  8a733cbd
06 Sep, 2023 1 commit
- [Rotary] Set device before launching Triton kernel to avoid error · 97951590
  Tri Dao authored Sep 05, 2023
  
  97951590
05 Sep, 2023 1 commit
- Create __init__.py for ops/triton dir (#516) · 8e893f09
  Kyeongpil Kang authored Sep 06, 2023
  
  8e893f09
04 Sep, 2023 1 commit
- [Rotary] Implement varlen rotary · b28ec236
  Tri Dao authored Sep 03, 2023
  
  b28ec236
03 Sep, 2023 3 commits
- [Rotary] Clean up rotary Triton implementation a bit · 861c8257
  Tri Dao authored Sep 03, 2023
  
  861c8257
- [Rotary] Speed up rotary kernel when interleaved=True · 1c523c1c
  Tri Dao authored Sep 03, 2023
  
  1c523c1c
- [Rotary] Implement rotary in Triton · 942fcbf0
  Tri Dao authored Sep 03, 2023
  
  942fcbf0
18 Aug, 2023 1 commit
- Run isort and black on python files · f1a73d07
  Tri Dao authored Aug 18, 2023
  
  f1a73d07
19 Apr, 2023 1 commit
- Implement LLaMa · 96d10f65
  Tri Dao authored Apr 18, 2023
  
  96d10f65
13 Apr, 2023 1 commit
- [FusedDense] Enable sqrelu activation in FusedMLP · 6f6e9a9a
  Tri Dao authored Apr 13, 2023
  
  6f6e9a9a
14 Nov, 2022 1 commit
- Add GPT and ViT models · 2e33fc8e
  Tri Dao authored Nov 13, 2022
  
  2e33fc8e