Commits · 717c5274462e171925129da8081fcd6e5904f3af · OpenDAS / Megatron-LM

06 Apr, 2023 14 commits
- Merge branch 'nextlm-merge-sentencepiece' into 'main' · 717c5274
  Jared Casper authored Apr 06, 2023
```
Add support for GPTSentencePieceTokenizer and related fixes.

See merge request ADLR/megatron-lm!561
```
  717c5274
- Merge branch 'untie_embeddings' into 'main' · 018391a6
  Jared Casper authored Apr 06, 2023
```
Untie Embeddings

See merge request ADLR/megatron-lm!558
```
  018391a6
- Missed some changes from next-best-lm branch. · 3207c19a
  Jared Casper authored Apr 06, 2023
  
  3207c19a
- Add support for GPTSentencePieceTokenizer and related fixes. · 46ffb75c
  Jared Casper authored Apr 06, 2023
  
  46ffb75c
- Merge branch 'next-best-lm/merge-layernorm-1p-main' into 'main' · 4e891fe9
  Jared Casper authored Apr 05, 2023
```
layernorm1p added

See merge request ADLR/megatron-lm!557
```
  4e891fe9
- Merge branch 'next-best-lm/merge-rope-main' into 'main' · 7bd25e26
  Jared Casper authored Apr 05, 2023
```
merging rope to main

See merge request ADLR/megatron-lm!556
```
  7bd25e26
- Add assert in BertModel that untie_embeddings_and_output_weights is False · 1e2a0405
  Jared Casper authored Apr 05, 2023
  
  1e2a0405
- Merge branch 'main' into untie_embeddings · c43d19ab
  Jared Casper authored Apr 05, 2023
  
  c43d19ab
- Merge branch 'main' into next-best-lm/merge-rope-main · 74bd02ec
  Jared Casper authored Apr 05, 2023
  
  74bd02ec
- Merge branch 'main' into next-best-lm/merge-layernorm-1p-main · 33a58153
  Jared Casper authored Apr 05, 2023
  
  33a58153
- Expand on apply-layernorm-1p description a bit. · 05b808ef
  Jared Casper authored Apr 05, 2023
  
  05b808ef
- A bit of cleanup. · 93bed794
  Jared Casper authored Apr 05, 2023
  
  93bed794
- Merge branch 'activations-bias' into 'main' · f6d36d03
  Jared Casper authored Apr 05, 2023
```
Add swiglu and squared relu activations and ability to disable bias.

See merge request ADLR/megatron-lm!553
```
  f6d36d03
- Merge branch 'bugFix' into 'main' · 22f054d8
  Shanmugam Ramasamy authored Apr 05, 2023
```
Update .gitlab-ci.yml

See merge request ADLR/megatron-lm!560
```
  22f054d8
05 Apr, 2023 4 commits
- Update .gitlab-ci.yml · 939383ae
  Shanmugam Ramasamy authored Apr 05, 2023
  
  939383ae
- Fix mpu init · 716da5d8
  MaximumEntropy authored Apr 05, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  716da5d8
- addressing the comments · 5ed304e4
  Mostofa Patwary authored Apr 05, 2023
  
  5ed304e4
- addressing the comments · 0760822b
  Mostofa Patwary authored Apr 05, 2023
  
  0760822b
04 Apr, 2023 5 commits
- Fix output layer init func when untie embeddings is True · 0f6bfdeb
  Sandeep Subramanian authored Feb 06, 2023
  
  0f6bfdeb
- Merge branch 'better-Output' into 'main' · 58020503
  Shanmugam Ramasamy authored Apr 04, 2023
```
Better output

See merge request ADLR/megatron-lm!559
```
  58020503
- Better output · 2221d882
  Shanmugam Ramasamy authored Apr 04, 2023
  
  2221d882
- removed self_attention_pos_emb · e1f1aa06
  Mostofa Patwary authored Apr 04, 2023
  
  e1f1aa06
- addressing comments · bf1da27e
  Mostofa Patwary authored Apr 04, 2023
  
  bf1da27e
03 Apr, 2023 14 commits
- addressing more comments · 5b7736ca
  Mostofa Patwary authored Apr 03, 2023
  
  5b7736ca
- addressing more comments · 07ffa1b7
  Mostofa Patwary authored Apr 03, 2023
  
  07ffa1b7
- remove unneccessary argument check · a9f6569d
  Jimmy Zhang authored Apr 03, 2023
  
  a9f6569d
- Fix spacing · 4fcb2f45
  MaximumEntropy authored Apr 03, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  4fcb2f45
- Merge branch 'untie_embeddings' of... · 1f8bc276
  MaximumEntropy authored Apr 03, 2023
```
Merge branch 'untie_embeddings' of https://gitlab-master.nvidia.com/ADLR/megatron-lm into untie_embeddings
```
  1f8bc276
- Undo parallel state changes · 96f4c5d2
  MaximumEntropy authored Apr 03, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  96f4c5d2
- assertion check for T5 and untied embeddings · 6b50a8c6
  Jimmy Zhang authored Apr 03, 2023
  
  6b50a8c6
- addressed the comments · df521589
  Mostofa Patwary authored Apr 03, 2023
  
  df521589
- Save word embeddings for head only only if embeddings are not untied · 18b26ec6
  Sandeep Subramanian authored Feb 15, 2023
  
  18b26ec6
- disable embedding addreduce if untie_embeddings_and_output_weights · f11b4c99
  Jimmy Zhang authored Jan 31, 2023
  
  f11b4c99
- Add assert for pp and untie · a3fbac58
  MaximumEntropy authored Dec 08, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  a3fbac58
- Fix checkpointing · 36c3a1c1
  MaximumEntropy authored Dec 08, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  36c3a1c1
- Initial commit for untied embeddings · 82c7ba57
  MaximumEntropy authored Dec 07, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  82c7ba57
- Addressing comments. · 420eec74
  Jared Casper authored Apr 03, 2023
  
  420eec74
02 Apr, 2023 1 commit
- layernorm1p added · 5e079c87
  Mostofa Patwary authored Apr 02, 2023
  
  5e079c87
01 Apr, 2023 2 commits
- Add swiglu and squared relu activations and ability to disable bias. · 2cc3dac7
  Jared Casper authored Apr 01, 2023
  
  2cc3dac7
- Merge branch 'core-pipeline-fix' into 'main' · fdd34a82
  Jared Casper authored Apr 01, 2023
```
Fix bug in core pipeline schedule

See merge request ADLR/megatron-lm!552
```
  fdd34a82