Commits · 3524e13c11c7866511973a39ebfd60ff6d74f8ce · gaoqiong / flash-attention

13 Aug, 2023 1 commit
- Update to Cutlass 3.1 · 3524e13c
  Tri Dao authored Aug 13, 2023
  
  3524e13c
01 Aug, 2023 2 commits
- Fix race condition in bwd (overwriting sK) · 1c41d2b0
  Tri Dao authored Aug 01, 2023
  
  1c41d2b0
- Fix masking of bwd when seqlen is not divisible by 128 · a4f148b6
  Tri Dao authored Jul 31, 2023
  
  a4f148b6
27 Jul, 2023 1 commit

Kirthi Shankar Sivamani authored Jul 27, 2023



* Add RNG state to kernel launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Save seed and offset for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Single thread write to global mem
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1colblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* compute_dq_dk_dv_1rowblock get seed and offset from launch params
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change forward c++ APIs to save RNG state for backward
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Change backward c++ APIs to set RNG state for bprop launcher
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Python side API changes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fix; only save seeds instead of full offset
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Account for 3D grid size
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

a03f6f8e

21 Jul, 2023 1 commit
- Fix using dO stride for O, which can cause memory error in bwd · 9ee0ff1d
  Tri Dao authored Jul 20, 2023
  
  9ee0ff1d
17 Jul, 2023 1 commit
- FlashAttention-2 release · 4f285b35
  Tri Dao authored Jul 17, 2023
  
  4f285b35