Commits · 7d25a4ec4f4ecef5df1a208ec74ca34455dcb439 · gaoqiong / flash-attention

13 Apr, 2023 4 commits
- Handle FlashAttnQKVPackedSplitFunc by making rng_state optional in backward · 7d25a4ec
  Kirthi Shankar Sivamani authored Apr 13, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  7d25a4ec
- Merge branch 'HazyResearch:main' into enable_cuda_graph_capture · 315fd31f
  Kirthi Shankar Sivamani authored Apr 12, 2023
  
  315fd31f
- Merge pull request #164 from ZhiyuanChen/patch-1 · 5cee0714
  Tri Dao authored Apr 12, 2023
```
make mlp hidden_features defaults to 4*in_features
```
  5cee0714
- make mlp hidden_features defaults to 4*in_features · 8c424156
  Zhiyuan Chen authored Apr 13, 2023
  
  8c424156
12 Apr, 2023 3 commits
- Support CUDA graph capture · 31018c5f
  Kirthi Shankar Sivamani authored Apr 12, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  31018c5f
- Bump version to v1.0.1, fix Cutlass version · 853ff729
  Tri Dao authored Apr 12, 2023
  
  853ff729
- Bump version to 1.0.0 · 74af0233
  Tri Dao authored Apr 11, 2023
  
  74af0233
07 Apr, 2023 1 commit
- [FusedDense] Set workspace size to 32M for Hopper and 4M for others · dec4f2e9
  Tri Dao authored Apr 06, 2023
  
  dec4f2e9
04 Apr, 2023 2 commits
- Merge pull request #154 from kuizhiqing/usage · d478eeec
  Tri Dao authored Apr 04, 2023
```
add paddlepaddle in usage
```
  d478eeec
- add paddlepaddle in usage · c5be8d3a
  kuizhiqing authored Apr 04, 2023
  
  c5be8d3a
31 Mar, 2023 3 commits
- Merge pull request #147 from ksivaman/add_deterministic_execution_option · d6fc8605
  Tri Dao authored Mar 31, 2023
```
Add option for deterministic execution
```
  d6fc8605
- [LayerNorm] Implement LN with parallel residual, support dim 8k · 393882bc
  Tri Dao authored Mar 29, 2023
  
  393882bc
- Add option for deterministic execution · b6aa059b
  Kirthi Shankar Sivamani authored Mar 30, 2023
  
  b6aa059b
29 Mar, 2023 3 commits
- [Training] Fix lightning _PATH import · 009a3e71
  Tri Dao authored Mar 29, 2023
  
  009a3e71
- Implement GPT-NeoX · 993d1244
  Tri Dao authored Mar 29, 2023
  
  993d1244
- [FT] Fix FT's single query attention for bf16 hdim128 rotary · f5d0fbd4
  Tri Dao authored Mar 28, 2023
  
  f5d0fbd4
22 Mar, 2023 3 commits
- Implement GPT-J · 4d87e4d8
  Tri Dao authored Mar 22, 2023
  
  4d87e4d8
- [Triton] Fix benchmark_causal.py · 4360cfc6
  Tri Dao authored Mar 22, 2023
  
  4360cfc6
- [Triton] Fix benchmark_causal, mention Triton version · 5d079fdd
  Tri Dao authored Mar 22, 2023
  
  5d079fdd
15 Mar, 2023 4 commits
- Support H100 for other CUDA extensions · dc08ea1c
  Tri Dao authored Mar 15, 2023
  
  dc08ea1c
- Support H100 · 1b18f1b7
  Tri Dao authored Mar 15, 2023
  
  1b18f1b7
- Merge pull request #140 from VikParuchuri/main · 318e2f1b
  Tri Dao authored Mar 15, 2023
```
Remove unused kwargs like device in FlashAttention
```
  318e2f1b
- Remove unused kwargs in flashattention · 31653980
  Vik Paruchuri authored Mar 15, 2023
  
  31653980
14 Mar, 2023 1 commit
- [Rotary] Implement GPT-J style (interleaved) rotary · e45a46a5
  Tri Dao authored Mar 14, 2023
  
  e45a46a5
13 Mar, 2023 1 commit
- Update README on requirements (nvcc and Pytorch) · f28d61cb
  Tri Dao authored Mar 13, 2023
  
  f28d61cb
15 Feb, 2023 1 commit
- Merge pull request #94 from calebthomas259/main · 57ee6181
  Tri Dao authored Feb 14, 2023
```
Add a simple tutorial to README.md
```
  57ee6181
09 Feb, 2023 1 commit
- Update roadmap · 2dc2a195
  Tri Dao authored Feb 09, 2023
  
  2dc2a195
27 Jan, 2023 1 commit
- Merge pull request #110 from eltociear/patch-1 · 06da275b
  Tri Dao authored Jan 27, 2023
```
fix typo in default.yaml
```
  06da275b
25 Jan, 2023 1 commit
- [FA] Remove unused variable rng_engine_inputs · 6b4a4821
  Tri Dao authored Jan 25, 2023
  
  6b4a4821
23 Jan, 2023 1 commit
- [OPT] Load fp16 weights on CPU before moving to GPU · 78b7a1dc
  Tri Dao authored Jan 22, 2023
  
  78b7a1dc
20 Jan, 2023 1 commit
- fix typo in default.yaml · 419ea45b
  Ikko Eltociear Ashimine authored Jan 21, 2023
```
additionaly -> additionally
```
  419ea45b
19 Jan, 2023 2 commits
- Bump to v0.2.8 · 33e0860c
  Tri Dao authored Jan 19, 2023
  
  33e0860c
- [LayerNorm] Rename x1 -> residual · eb33e587
  Tri Dao authored Jan 19, 2023
  
  eb33e587
18 Jan, 2023 2 commits
- [Gen] Add OPT to generation test · f68d41ec
  Tri Dao authored Jan 17, 2023
  
  f68d41ec
- [FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP · 88173a1a
  Tri Dao authored Jan 17, 2023
  
  88173a1a
16 Jan, 2023 4 commits
- [ViT] Support timm checkpoint, add tests · 780e8eea
  Tri Dao authored Jan 16, 2023
  
  780e8eea
- Merge pull request #105 from jamaliki/patch-1 · 2ec7d3f7
  Tri Dao authored Jan 15, 2023
```
Change default dropout value in documentation
```
  2ec7d3f7
- [ViT] Fix extra norm_0, use new LN order in Block · ef085cfc
  Tri Dao authored Jan 15, 2023
  
  ef085cfc
- Reorder LN in Block, support OPT · ff34123b
  Tri Dao authored Jan 15, 2023
  
  ff34123b
15 Jan, 2023 1 commit
- [Gen] Pass qkv_stride to ft_attention kernel for batched generation · f1e01c27
  Tri Dao authored Jan 15, 2023
  
  f1e01c27