Commits · dc72d960a7fc903ee34b8df84a056620b354ec47 · gaoqiong / flash-attention

"langchain_demo/Tool/arxiv_example.yaml" did not exist on "467ec853e060e460d555769731d48652c9df3a91"

05 Jan, 2024 1 commit
- [LayerNorm] Switch from CUDA to Triton implementation · abbc1311
  Tri Dao authored Jan 05, 2024
  
  abbc1311
25 Dec, 2023 2 commits
- Add test for BTLM init · 73df3be7
  Tri Dao authored Dec 25, 2023
  
  73df3be7
- Implement muParam · 2e29dacf
  Tri Dao authored Dec 24, 2023
  
  2e29dacf
23 Dec, 2023 1 commit
- Implement norm head for Baichuan2 · 2c7d7b73
  Tri Dao authored Dec 22, 2023
  
  2c7d7b73
22 Dec, 2023 1 commit
- Add Alibi to MHA, test with Baichuan-13B · c3b21966
  Tri Dao authored Dec 21, 2023
  
  c3b21966
21 Sep, 2023 1 commit
- Fix E1136 (#563) · 187c2a06
  Yuchao Dai authored Sep 22, 2023
  
  187c2a06
13 Sep, 2023 1 commit
- Add tests for Pythia, GPT-JT, and RedPajama models · d0032700
  Tri Dao authored Sep 13, 2023
  
  d0032700
11 Sep, 2023 1 commit
- Add BigCode converters (#532) · 07005806
  Kevin Hu authored Sep 10, 2023
  
  07005806
04 Sep, 2023 2 commits
- Fix test_baichuan · 798858f9
  Tri Dao authored Sep 03, 2023
  
  798858f9
- [Gen] Add back num_last_tokens in gpt.py · 7b33743a
  Tri Dao authored Sep 03, 2023
  
  7b33743a
30 Aug, 2023 1 commit
- Support MQA + MP for decoding (#490) · 011ec323
  dan_the_3rd authored Aug 30, 2023
```
Co-authored-by: danthe3rd <danthe3rd>
```
  011ec323
27 Aug, 2023 1 commit
- [GPT] Generalize last_token_only arg to num_last_tokens · f8aea6ea
  Tri Dao authored Aug 26, 2023
  
  f8aea6ea
24 Aug, 2023 1 commit
- add llama support to GPTPreTrainedModel.from_pretrained (#479) · e0b09891
  Aman Gupta Karmani authored Aug 24, 2023
  
  e0b09891
20 Aug, 2023 1 commit
- handle uneven heads across ranks when combining state_dicts; resolves #467 (#468) · 25d6b1db
  Xuechen Li authored Aug 20, 2023
```
* q

* add comment.
```
  25d6b1db
19 Aug, 2023 1 commit

map custom model state_dict back to huggingface format (#465) · 7fcd3e6a

Xuechen Li authored Aug 18, 2023

* fix name.

* set inv function.

* add map back function.

* handle gqa.

* add type annotation to avoid confusion.

* fix docstr.

* test inverse remap logic.

7fcd3e6a

18 Aug, 2023 2 commits

Run isort and black on python files · f1a73d07
Tri Dao authored Aug 18, 2023

f1a73d07

support when num_heads is not divisible by world_size; resolves #459 (#461) · bb4cded1

Xuechen Li authored Aug 18, 2023

* uneql rank.

* trim.

* enable passing in number of heads for each rank.

* simplify.

* simplify.

* cleanup.

* fix col parallel.

* fix bug with row parallel.

* fit out proj.

* refac.

* fix sharding logic.

* refac sharding.

* refac.

* support multiple of.

* make fn reuseable.

* fix bug in dimensions.

* scaffold.

* test uneven heads.

* fix test by adding barrier.

* refac.

* reuse code.

* clean up.

bb4cded1

17 Aug, 2023 1 commit
- [GPT] Run black on gpt.py · 4b661a56
  Tri Dao authored Aug 16, 2023
  
  4b661a56
29 Jul, 2023 1 commit
- [GPT] Implement parallel LLaMa · 184b992d
  Tri Dao authored Jul 28, 2023
  
  184b992d
26 Jul, 2023 1 commit
- Implement ParallelGatedMlp (#251) · 8ee62efc
  Haodong Lyu authored Jul 27, 2023
  
  8ee62efc
23 Jul, 2023 4 commits
- [GPT] Implement Falcon · d38357dd
  Tri Dao authored Jul 23, 2023
  
  d38357dd
- [MHA] Implement MQA/GQA · 425dbcb6
  Tri Dao authored Jul 23, 2023
  
  425dbcb6
- [Rotary] Don't store inv_freq in state_dict · ec9f74ab
  Tri Dao authored Jul 22, 2023
  
  ec9f74ab
- [MLP] Add ParallelMLP · 75e334d4
  Tri Dao authored Jul 22, 2023
  
  75e334d4
30 May, 2023 1 commit
- [Gen] Add rotary base as an argument to FT attention kernel · 48bc6eac
  Tri Dao authored May 30, 2023
  
  48bc6eac
06 May, 2023 1 commit
- [BugFix] cannot unpack non-iterable NoneType object · 69f5f7d0
  Federico Berto authored May 07, 2023
  
  69f5f7d0
05 May, 2023 1 commit
- [LLaMa] Fix last norm layer to use RMSNorm instead of LayerNorm · a9a4b4e4
  Tri Dao authored May 04, 2023
  
  a9a4b4e4
21 Apr, 2023 2 commits
- [Gen] Move allocate_inference_cache to within the model · ba2fe7f3
  Tri Dao authored Apr 20, 2023
  
  ba2fe7f3
- [GPT] Add option to only return the logit for the last token · 3da42d24
  Tri Dao authored Apr 20, 2023
  
  3da42d24
19 Apr, 2023 1 commit
- Implement LLaMa · 96d10f65
  Tri Dao authored Apr 18, 2023
  
  96d10f65
18 Apr, 2023 1 commit
- Implement GatedMlp · b630aef5
  Tri Dao authored Apr 18, 2023
  
  b630aef5
13 Apr, 2023 1 commit
- [FusedDense] Enable sqrelu activation in FusedMLP · 6f6e9a9a
  Tri Dao authored Apr 13, 2023
  
  6f6e9a9a
31 Mar, 2023 1 commit
- [LayerNorm] Implement LN with parallel residual, support dim 8k · 393882bc
  Tri Dao authored Mar 29, 2023
  
  393882bc
29 Mar, 2023 1 commit
- Implement GPT-NeoX · 993d1244
  Tri Dao authored Mar 29, 2023
  
  993d1244
22 Mar, 2023 1 commit
- Implement GPT-J · 4d87e4d8
  Tri Dao authored Mar 22, 2023
  
  4d87e4d8
23 Jan, 2023 1 commit
- [OPT] Load fp16 weights on CPU before moving to GPU · 78b7a1dc
  Tri Dao authored Jan 22, 2023
  
  78b7a1dc
19 Jan, 2023 1 commit
- [LayerNorm] Rename x1 -> residual · eb33e587
  Tri Dao authored Jan 19, 2023
  
  eb33e587
18 Jan, 2023 1 commit
- [FusedDense] Support relu, rename FusedDenseGeluDense -> FusedMLP · 88173a1a
  Tri Dao authored Jan 17, 2023
  
  88173a1a
16 Jan, 2023 1 commit
- Reorder LN in Block, support OPT · ff34123b
  Tri Dao authored Jan 15, 2023
  
  ff34123b
15 Jan, 2023 1 commit
- [Gen] Make generation work with Tensor Parallel · 7c219154
  Tri Dao authored Jan 15, 2023
  
  7c219154