Commits · e07aa036db0e6cd251b17d9ee2d1720023da2def · gaoqiong / flash-attention

24 Aug, 2023 1 commit
- Support flash attention 2 with causal masking when KV's seq length is longer... · e07aa036
  BoxiangW authored Aug 24, 2023
```
Support flash attention 2 with causal masking when KV's seq length is longer than Q's seq length. (#436)
```
  e07aa036
17 Aug, 2023 1 commit
- [FusedDense] Run black on fused_dense.py · bcfa7c97
  Tri Dao authored Aug 16, 2023
  
  bcfa7c97
16 Aug, 2023 1 commit
- Fix Bwd NaN for varlen when seqlen_q >> seqlen_k and causal · c65b5106
  Tri Dao authored Aug 16, 2023
  
  c65b5106
13 Aug, 2023 2 commits
- Prepare for Cutlass 3.2 · dbd79237
  Tri Dao authored Aug 13, 2023
  
  dbd79237
- Update to Cutlass 3.1 · 3524e13c
  Tri Dao authored Aug 13, 2023
  
  3524e13c
01 Aug, 2023 2 commits
- Fix race condition in bwd (overwriting sK) · 1c41d2b0
  Tri Dao authored Aug 01, 2023
  
  1c41d2b0
- Fix masking of bwd when seqlen is not divisible by 128 · a4f148b6
  Tri Dao authored Jul 31, 2023
  
  a4f148b6
27 Jul, 2023 1 commit

Kirthi Shankar Sivamani authored Jul 27, 2023



* Add RNG state to kernel launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Save seed and offset for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Single thread write to global mem
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1colblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1rowblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change forward c++ APIs to save RNG state for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change backward c++ APIs to set RNG state for bprop launcher
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Python side API changes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fix; only save seeds instead of full offset
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Account for 3D grid size
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

a03f6f8e

23 Jul, 2023 2 commits
- Fix random state for dropout_layer_norm (#315) · 767b71cc
  Joel Lamy-Poirier authored Jul 23, 2023
  
  767b71cc
- [FT] Implement MQA/GQA · a157cc8c
  Tri Dao authored Jul 22, 2023
  
  a157cc8c
21 Jul, 2023 1 commit
- Fix using dO stride for O, which can cause memory error in bwd · 9ee0ff1d
  Tri Dao authored Jul 20, 2023
  
  9ee0ff1d
19 Jul, 2023 2 commits
- [LayerNorm] Fix typo in ln_api.cpp · dfc60f6b
  Ikko Eltociear Ashimine authored Jul 20, 2023
```
unintialized -> uninitialized
```
  dfc60f6b
- Fix compile error on MSVC · 538d570c
  danthe3rd authored Jul 19, 2023
```
See also: https://stackoverflow.com/questions/55136414/constexpr-variable-captured-inside-lambda-loses-its-constexpr-ness
```
  538d570c
17 Jul, 2023 1 commit
- FlashAttention-2 release · 4f285b35
  Tri Dao authored Jul 17, 2023
  
  4f285b35
06 Jul, 2023 1 commit
- [FT] rotary_cos/sin should have batch_size dimension · 2800efc7
  Tri Dao authored Jul 06, 2023
  
  2800efc7
03 Jul, 2023 1 commit
- [FT] rotary_cos/sin should have shape (dim) instead of (seqlen, dim) · 3a9bfd07
  Tri Dao authored Jul 03, 2023
  
  3a9bfd07
02 Jul, 2023 1 commit
- [Rotary] Make sure frequency calculation is in fp32 · 62e98144
  Tri Dao authored Jul 02, 2023
  
  62e98144
30 May, 2023 2 commits
- [FusedDense] Allocate lt_workspace on input device · 27f8f890
  Tri Dao authored May 30, 2023
  
  27f8f890
- [Gen] Add rotary base as an argument to FT attention kernel · 48bc6eac
  Tri Dao authored May 30, 2023
  
  48bc6eac
26 Apr, 2023 1 commit
- [Docs] Clearer error message for bwd d > 64, bump to v1.0.4 · ad113948
  Tri Dao authored Apr 26, 2023
  
  ad113948
21 Apr, 2023 1 commit
- [Gen] Fix FT kernel smem size, CG when batch size changed · 311d6606
  Tri Dao authored Apr 20, 2023
  
  311d6606
15 Apr, 2023 1 commit
- only 1 thread writes to global mem in fprop · 45567a25
  Kirthi Shankar Sivamani authored Apr 15, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  45567a25
13 Apr, 2023 1 commit
- Handle FlashAttnQKVPackedSplitFunc by making rng_state optional in backward · 7d25a4ec
  Kirthi Shankar Sivamani authored Apr 13, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  7d25a4ec
12 Apr, 2023 1 commit
- Support CUDA graph capture · 31018c5f
  Kirthi Shankar Sivamani authored Apr 12, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  31018c5f
07 Apr, 2023 1 commit
- [FusedDense] Set workspace size to 32M for Hopper and 4M for others · dec4f2e9
  Tri Dao authored Apr 06, 2023
  
  dec4f2e9
31 Mar, 2023 1 commit
- [LayerNorm] Implement LN with parallel residual, support dim 8k · 393882bc
  Tri Dao authored Mar 29, 2023
  
  393882bc
29 Mar, 2023 1 commit
- [FT] Fix FT's single query attention for bf16 hdim128 rotary · f5d0fbd4
  Tri Dao authored Mar 28, 2023
  
  f5d0fbd4
15 Mar, 2023 2 commits
- Support H100 for other CUDA extensions · dc08ea1c
  Tri Dao authored Mar 15, 2023
  
  dc08ea1c
- Support H100 · 1b18f1b7
  Tri Dao authored Mar 15, 2023
  
  1b18f1b7
14 Mar, 2023 1 commit
- [Rotary] Implement GPT-J style (interleaved) rotary · e45a46a5
  Tri Dao authored Mar 14, 2023
  
  e45a46a5
25 Jan, 2023 1 commit
- [FA] Remove unused variable rng_engine_inputs · 6b4a4821
  Tri Dao authored Jan 25, 2023
  
  6b4a4821
19 Jan, 2023 1 commit
- [LayerNorm] Rename x1 -> residual · eb33e587
  Tri Dao authored Jan 19, 2023
  
  eb33e587
18 Jan, 2023 1 commit
- [FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP · 88173a1a
  Tri Dao authored Jan 17, 2023
  
  88173a1a
15 Jan, 2023 2 commits
- [Gen] Pass qkv_stride to ft_attention kernel for batched generation · f1e01c27
  Tri Dao authored Jan 15, 2023
  
  f1e01c27
- [Gen] Make generation work with Tensor Parallel · 7c219154
  Tri Dao authored Jan 15, 2023
  
  7c219154
07 Jan, 2023 1 commit
- [LayerNorm] Implement RMS Norm · 6738d947
  Tri Dao authored Jan 06, 2023
  
  6738d947
06 Jan, 2023 1 commit
- [Compilation] Change BOOL_SWITCH to fix Windows compilation · a1f49a2b
  Tri Dao authored Jan 06, 2023
```
Follow xFormers's DISTPATCH_BOOL. Haven't tested it on Windows.
```
  a1f49a2b
04 Jan, 2023 3 commits
- [Gen, FT] Use fp32 accum for FMA · be1afaa2
  Tri Dao authored Jan 03, 2023
  
  be1afaa2
- [Gen, FT] Use tlength instead of params.timestep for rotary · f266fc72
  Tri Dao authored Jan 03, 2023
  
  f266fc72
- [Gen] Add kernel from FasterTransformer for benchmarking · a01d1213
  Tri Dao authored Jan 03, 2023
  
  a01d1213