Commits · ce26d3d73d07e9779c5ba6fb2ca3bc187a34c6cc · gaoqiong / flash-attention

07 Jan, 2023 1 commit
- [LayerNorm] Implement RMS Norm · 6738d947
  Tri Dao authored Jan 06, 2023
  
  6738d947
04 Jan, 2023 1 commit
- [Gen] Add option to run generation with FT attention kernel · a668890f
  Tri Dao authored Jan 03, 2023
  
  a668890f
01 Jan, 2023 1 commit
- [GPT] Refactor function to shard state_dict for TensorParallel · ef1ba918
  Tri Dao authored Jan 01, 2023
  
  ef1ba918
28 Dec, 2022 1 commit
- Implement generation for GPT · 63670fd8
  Tri Dao authored Dec 27, 2022
  
  63670fd8
27 Dec, 2022 3 commits
- Support loading GPT2 weights from Huggingface · 9d797d88
  Tri Dao authored Dec 27, 2022
  
  9d797d88
- Tweak CrossEntropyLoss to take process_group in init · c6ecd40a
  Tri Dao authored Dec 27, 2022
  
  c6ecd40a
- Implement Tensor Parallel for GPT model · b4018a50
  Tri Dao authored Dec 25, 2022
  
  b4018a50
25 Dec, 2022 3 commits
- Implement Tensor Parallel for GPT2Embeddings · 78225c53
  Tri Dao authored Dec 25, 2022
  
  78225c53
- Implement Tensor Parallel for transformer Block · a8cfe515
  Tri Dao authored Dec 25, 2022
  
  a8cfe515
- Implement TensorParallel for MHA · 1e712ea8
  Tri Dao authored Dec 24, 2022
  
  1e712ea8
24 Dec, 2022 1 commit
- Implement TensorParallel for FusedDense and FusedDenseGeluDense · 226a1b72
  Tri Dao authored Dec 23, 2022
  
  226a1b72
23 Dec, 2022 2 commits
- Add smoothing for CrossEntropyParallel, rename to CrossEntropyLoss · dff68c2b
  Tri Dao authored Dec 23, 2022
  
  dff68c2b
- Simplify FusedDense · e68ebbe8
  Tri Dao authored Dec 22, 2022
  
  e68ebbe8
20 Dec, 2022 1 commit
- Implement last_layer_subset optimization for BERT · 13cdceb3
  Tri Dao authored Dec 19, 2022
  
  13cdceb3
19 Dec, 2022 1 commit
- Implement BERT · 5fb6df0e
  Tri Dao authored Dec 18, 2022
  
  5fb6df0e
13 Dec, 2022 1 commit
- [LayerNorm] Support taking subset of input or subset of output · 5db33051
  Tri Dao authored Dec 12, 2022
  
  5db33051
11 Dec, 2022 1 commit
- [LayerNorm] Fuse LayerScale · ae137ed1
  Tri Dao authored Dec 10, 2022
  
  ae137ed1
09 Dec, 2022 1 commit
- [LayerNorm] Support all dimensions up to 6k (if divisible by 8) · 8c6609ae
  Tri Dao authored Dec 08, 2022
  
  8c6609ae
14 Nov, 2022 3 commits
- Add MLP, MHA, Block, Embedding modules · d4b320b3
  Tri Dao authored Nov 13, 2022
  
  d4b320b3
- Add fused_dense and dropout_add_layernorm CUDA extensions · fa6d1ce4
  Tri Dao authored Nov 13, 2022
  
  fa6d1ce4
- Make nccl operations async in CrossEntropyLossParallel · 343492ec
  Tri Dao authored Nov 13, 2022
  
  343492ec
13 Nov, 2022 3 commits
- Skip flash_attn_split test · a8fec99a
  Tri Dao authored Nov 13, 2022
  
  a8fec99a
- Don't enforce bitwise consistency for dq in race condition test · 9d3116ad
  Tri Dao authored Nov 13, 2022
```
Since we could be parallelizing over seqlen_k
```
  9d3116ad
- Add fused cross entropy loss · 7c995381
  Tri Dao authored Nov 12, 2022
  
  7c995381
09 Nov, 2022 1 commit
- Fix out-of-bound memory read · 6998e0ec
  Tri Dao authored Nov 09, 2022
  
  6998e0ec
06 Nov, 2022 1 commit
- Fix pipelining bug in Triton bwd with bias_type=matrix · 74797571
  Tri Dao authored Nov 06, 2022
  
  74797571
05 Nov, 2022 2 commits
- Parallelize CUDA bwd along seqlen_k instead of seqlen_q · 55778193
  Tri Dao authored Nov 05, 2022
```
This is faster since we only need to do atomic adds on dq, instead of atomic
adds on both dk and dv.
```
  55778193
- Implement rotary embedding in CUDA · ca81f32e
  Tri Dao authored Nov 04, 2022
  
  ca81f32e
04 Nov, 2022 2 commits
- Fix race condition in Triton bwd when there's bias · ff78ea41
  Tri Dao authored Nov 04, 2022
  
  ff78ea41
- Implement attention bias for Triton version · 86862cfd
  Tri Dao authored Nov 04, 2022
  
  86862cfd
02 Nov, 2022 1 commit
- Fix race condition in Triton bwd for non-po2 headdims · aacc10fb
  Tri Dao authored Nov 02, 2022
  
  aacc10fb
01 Nov, 2022 1 commit
- Avoid memcpy in the Triton bwd · 1fb12afd
  Tri Dao authored Nov 01, 2022
  
  1fb12afd
31 Oct, 2022 8 commits
- Fix race condition in Triton fwd · 9b0bc978
  Tri Dao authored Oct 31, 2022
  
  9b0bc978
- Add debug_barrier for all headdims in Triton bwd · 4f81aff4
  Tri Dao authored Oct 31, 2022
  
  4f81aff4
- [WIP] Support all head dimensions up to 128 in the Triton bwd · e78d509c
  Tri Dao authored Oct 31, 2022
```
WIP because there seems to be some race conditions for head dimensions other
than 16, 32, 64, 128.
```
  e78d509c
- Support all head dimensions up to 128 in the Triton fwd · 008951f1
  Tri Dao authored Oct 30, 2022
  
  008951f1
- Support arbitrary seqlens (both q & k) in Triton bwd · b910bf14
  Tri Dao authored Oct 30, 2022
  
  b910bf14
- Support arbitrary seqlen_k in Triton bwd · dc554693
  Tri Dao authored Oct 30, 2022
  
  dc554693
- Fix Triton fwd to support seqlen not multiples of 128 · d11341fd
  Tri Dao authored Oct 30, 2022
  
  d11341fd
- Implement FlashAttention in Triton · b0c0db81
  Tri Dao authored Oct 30, 2022
  
  b0c0db81