Commits · 8dbd075743a185547b9eb1491771df754757de1b · OpenDAS / Megatron-LM

07 Apr, 2023 2 commits
- add needed helper functions · 3ce6a1c2
  Abhinav Khattar authored Apr 07, 2023
```
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
```
  3ce6a1c2
- Some quick fixes. · 45084d46
  Jared Casper authored Apr 06, 2023
  
  45084d46
06 Apr, 2023 8 commits
- dropout variable · a7cce320
  Jimmy Zhang authored Apr 06, 2023
  
  a7cce320
- refactor flash attention · 254e8815
  Jimmy Zhang authored Apr 06, 2023
  
  254e8815
- Flash Attention inference fix · f1a50a3c
  Jimmy Zhang authored Mar 03, 2023
  
  f1a50a3c
- Missed some changes from next-best-lm branch. · 3207c19a
  Jared Casper authored Apr 06, 2023
  
  3207c19a
- Add support for GPTSentencePieceTokenizer and related fixes. · 46ffb75c
  Jared Casper authored Apr 06, 2023
  
  46ffb75c
- Add assert in BertModel that untie_embeddings_and_output_weights is False · 1e2a0405
  Jared Casper authored Apr 05, 2023
  
  1e2a0405
- Expand on apply-layernorm-1p description a bit. · 05b808ef
  Jared Casper authored Apr 05, 2023
  
  05b808ef
- A bit of cleanup. · 93bed794
  Jared Casper authored Apr 05, 2023
  
  93bed794
05 Apr, 2023 3 commits
- Fix mpu init · 716da5d8
  MaximumEntropy authored Apr 05, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  716da5d8
- addressing the comments · 5ed304e4
  Mostofa Patwary authored Apr 05, 2023
  
  5ed304e4
- addressing the comments · 0760822b
  Mostofa Patwary authored Apr 05, 2023
  
  0760822b
04 Apr, 2023 3 commits
- Fix output layer init func when untie embeddings is True · 0f6bfdeb
  Sandeep Subramanian authored Feb 06, 2023
  
  0f6bfdeb
- removed self_attention_pos_emb · e1f1aa06
  Mostofa Patwary authored Apr 04, 2023
  
  e1f1aa06
- addressing comments · bf1da27e
  Mostofa Patwary authored Apr 04, 2023
  
  bf1da27e
03 Apr, 2023 13 commits
- addressing more comments · 5b7736ca
  Mostofa Patwary authored Apr 03, 2023
  
  5b7736ca
- addressing more comments · 07ffa1b7
  Mostofa Patwary authored Apr 03, 2023
  
  07ffa1b7
- remove unneccessary argument check · a9f6569d
  Jimmy Zhang authored Apr 03, 2023
  
  a9f6569d
- Fix spacing · 4fcb2f45
  MaximumEntropy authored Apr 03, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  4fcb2f45
- Undo parallel state changes · 96f4c5d2
  MaximumEntropy authored Apr 03, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  96f4c5d2
- assertion check for T5 and untied embeddings · 6b50a8c6
  Jimmy Zhang authored Apr 03, 2023
  
  6b50a8c6
- addressed the comments · df521589
  Mostofa Patwary authored Apr 03, 2023
  
  df521589
- Save word embeddings for head only only if embeddings are not untied · 18b26ec6
  Sandeep Subramanian authored Feb 15, 2023
  
  18b26ec6
- disable embedding addreduce if untie_embeddings_and_output_weights · f11b4c99
  Jimmy Zhang authored Jan 31, 2023
  
  f11b4c99
- Add assert for pp and untie · a3fbac58
  MaximumEntropy authored Dec 08, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  a3fbac58
- Fix checkpointing · 36c3a1c1
  MaximumEntropy authored Dec 08, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  36c3a1c1
- Initial commit for untied embeddings · 82c7ba57
  MaximumEntropy authored Dec 07, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  82c7ba57
- Addressing comments. · 420eec74
  Jared Casper authored Apr 03, 2023
  
  420eec74
02 Apr, 2023 1 commit
- layernorm1p added · 5e079c87
  Mostofa Patwary authored Apr 02, 2023
  
  5e079c87
01 Apr, 2023 2 commits
- Add swiglu and squared relu activations and ability to disable bias. · 2cc3dac7
  Jared Casper authored Apr 01, 2023
  
  2cc3dac7
- Fix bug in core pipeline schedule · ff7b28f4
  Jared Casper authored Apr 01, 2023
  
  ff7b28f4
31 Mar, 2023 2 commits
- merging rope to main · 1e0e555c
  Mostofa Patwary authored Mar 31, 2023
  
  1e0e555c
- fix bug in uniform activation recompute · fc2c81d3
  Jared Casper authored Mar 31, 2023
  
  fc2c81d3
23 Mar, 2023 2 commits
- Move pipeline parallel functionality into core with associated changes. · 3c92fa93
  Jared Casper authored Mar 23, 2023
  
  3c92fa93
- Add contiguous to grad_output in layers.py · 09c2f6e4
  Markel Ausin authored Mar 22, 2023
  
  09c2f6e4
22 Feb, 2023 1 commit
- Retro · 17a6044d
  Lawrence McAfee authored Feb 21, 2023
  
  17a6044d
04 Jan, 2023 1 commit
- Remove FA's check for headdim <= 128 · 9200e43a
  Tri Dao authored Jan 03, 2023
  
  9200e43a
22 Dec, 2022 1 commit
- Transformer Engine Integration Rebase · 3499542e
  John Kamalu authored Dec 21, 2022
  
  3499542e
09 Dec, 2022 1 commit
- Integrate FlashAttention into Megatron-LM · d693034e
  Tri Dao authored Dec 08, 2022
  
  d693034e