Commits · 6738d9477dc906ec6089b44db2c4de8cb2923ee9 · gaoqiong / flash-attention

07 Jan, 2023 1 commit
- [LayerNorm] Implement RMS Norm · 6738d947
  Tri Dao authored Jan 06, 2023
  
  6738d947
02 Jan, 2023 1 commit
- [FusedDense] Limit matrix dims to 2M (instead of 64k) · 1ec09ebd
  Tri Dao authored Jan 01, 2023
  
  1ec09ebd
01 Jan, 2023 1 commit
- [FusedDense] Kick off input all_gather before weight dtype conversion · 65b4064b
  Tri Dao authored Dec 31, 2022
  
  65b4064b
25 Dec, 2022 1 commit
- Implement Tensor Parallel for transformer Block · a8cfe515
  Tri Dao authored Dec 25, 2022
  
  a8cfe515
24 Dec, 2022 1 commit
- Implement TensorParallel for FusedDense and FusedDenseGeluDense · 226a1b72
  Tri Dao authored Dec 23, 2022
  
  226a1b72
23 Dec, 2022 1 commit
- Simplify FusedDense · e68ebbe8
  Tri Dao authored Dec 22, 2022
  
  e68ebbe8
19 Dec, 2022 1 commit
- Implement BERT · 5fb6df0e
  Tri Dao authored Dec 18, 2022
  
  5fb6df0e
13 Dec, 2022 1 commit
- [LayerNorm] Support taking subset of input or subset of output · 5db33051
  Tri Dao authored Dec 12, 2022
  
  5db33051
11 Dec, 2022 1 commit
- [LayerNorm] Fuse LayerScale · ae137ed1
  Tri Dao authored Dec 10, 2022
  
  ae137ed1
09 Dec, 2022 1 commit
- [LayerNorm] Support all dimensions up to 6k (if divisible by 8) · 8c6609ae
  Tri Dao authored Dec 08, 2022
  
  8c6609ae
18 Nov, 2022 1 commit
- Add __init__.py files to subdirectories for installation · ece539ab
  Tri Dao authored Nov 17, 2022
  
  ece539ab
14 Nov, 2022 3 commits
- Add GPT and ViT models · 2e33fc8e
  Tri Dao authored Nov 13, 2022
  
  2e33fc8e
- Add MLP, MHA, Block, Embedding modules · d4b320b3
  Tri Dao authored Nov 13, 2022
  
  d4b320b3
- Add fused_dense and dropout_add_layernorm CUDA extensions · fa6d1ce4
  Tri Dao authored Nov 13, 2022
  
  fa6d1ce4