Commits · 3db2063b1ff992a971ba18f7101eecc9c4e90f03 · OpenDAS / Megatron-LM

19 Apr, 2023 8 commits
- Merge branch 'enable_autocast' into 'main' · 3db2063b
  Jared Casper authored Apr 19, 2023
```
Add ability to enable autocast through schedules

See merge request ADLR/megatron-lm!569
```
  3db2063b
- address review comments · 98bb40d9
  Abhinav Khattar authored Apr 19, 2023
```
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
```
  98bb40d9
- keep previous autocast check · 630811b2
  Abhinav Khattar authored Apr 19, 2023
```
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
```
  630811b2
- add enable autocast · 513d7d8e
  Abhinav Khattar authored Apr 19, 2023
```
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
```
  513d7d8e
- Merge branch 'jbarker/apex_layernorm' into 'main' · 2699f93e
  Jared Casper authored Apr 19, 2023
```
replace custom layer_norm_cuda with Apex layer_norm_cuda

See merge request ADLR/megatron-lm!566
```
  2699f93e
- replace custom layer_norm_cuda with Apex layer_norm_cuda · d650e6a2
  Jon Barker authored Apr 19, 2023
  
  d650e6a2
- Merge branch 'checkpoint_util_fix' into 'main' · 378db1cd
  Jared Casper authored Apr 19, 2023
```
Some quick fixes to checkpoint_util.

See merge request ADLR/megatron-lm!568
```
  378db1cd
- Some quick fixes to checkpoint_util. · be1a575e
  Jared Casper authored Apr 19, 2023
  
  be1a575e
14 Apr, 2023 1 commit
- Merge branch 'test-update' into 'main' · 8dbd0757
  Jared Casper authored Apr 13, 2023
```
Update tests with new gold numbers

See merge request ADLR/megatron-lm!565
```
  8dbd0757
13 Apr, 2023 1 commit
- Recent commit changed the bias dropout add jitted function which slightly... · 08e62070
  Jared Casper authored Apr 13, 2023
```
Recent commit changed the bias dropout add jitted function which slightly changes numbers, manually tested to be accurate.
```
  08e62070
09 Apr, 2023 1 commit
- Merge branch 'add_helper_fns' into 'main' · ee084835
  Jared Casper authored Apr 08, 2023
```
Add parallel_state helper functions

See merge request ADLR/megatron-lm!564
```
  ee084835
07 Apr, 2023 4 commits
- add needed helper functions · 3ce6a1c2
  Abhinav Khattar authored Apr 07, 2023
```
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
```
  3ce6a1c2
- Merge branch 'nextlm-fixes' into 'main' · 9c5e287d
  Jared Casper authored Apr 06, 2023
```
Some quick fixes.

See merge request ADLR/megatron-lm!563
```
  9c5e287d
- Some quick fixes. · 45084d46
  Jared Casper authored Apr 06, 2023
  
  45084d46
- Merge branch 'flash_attention_inference' into 'main' · cdd2afdf
  Jared Casper authored Apr 06, 2023
```
Flash Attention inference fix

See merge request ADLR/megatron-lm!562
```
  cdd2afdf
06 Apr, 2023 17 commits
- dropout variable · a7cce320
  Jimmy Zhang authored Apr 06, 2023
  
  a7cce320
- refactor flash attention · 254e8815
  Jimmy Zhang authored Apr 06, 2023
  
  254e8815
- Flash Attention inference fix · f1a50a3c
  Jimmy Zhang authored Mar 03, 2023
  
  f1a50a3c
- Merge branch 'nextlm-merge-sentencepiece' into 'main' · 717c5274
  Jared Casper authored Apr 06, 2023
```
Add support for GPTSentencePieceTokenizer and related fixes.

See merge request ADLR/megatron-lm!561
```
  717c5274
- Merge branch 'untie_embeddings' into 'main' · 018391a6
  Jared Casper authored Apr 06, 2023
```
Untie Embeddings

See merge request ADLR/megatron-lm!558
```
  018391a6
- Missed some changes from next-best-lm branch. · 3207c19a
  Jared Casper authored Apr 06, 2023
  
  3207c19a
- Add support for GPTSentencePieceTokenizer and related fixes. · 46ffb75c
  Jared Casper authored Apr 06, 2023
  
  46ffb75c
- Merge branch 'next-best-lm/merge-layernorm-1p-main' into 'main' · 4e891fe9
  Jared Casper authored Apr 05, 2023
```
layernorm1p added

See merge request ADLR/megatron-lm!557
```
  4e891fe9
- Merge branch 'next-best-lm/merge-rope-main' into 'main' · 7bd25e26
  Jared Casper authored Apr 05, 2023
```
merging rope to main

See merge request ADLR/megatron-lm!556
```
  7bd25e26
- Add assert in BertModel that untie_embeddings_and_output_weights is False · 1e2a0405
  Jared Casper authored Apr 05, 2023
  
  1e2a0405
- Merge branch 'main' into untie_embeddings · c43d19ab
  Jared Casper authored Apr 05, 2023
  
  c43d19ab
- Merge branch 'main' into next-best-lm/merge-rope-main · 74bd02ec
  Jared Casper authored Apr 05, 2023
  
  74bd02ec
- Merge branch 'main' into next-best-lm/merge-layernorm-1p-main · 33a58153
  Jared Casper authored Apr 05, 2023
  
  33a58153
- Expand on apply-layernorm-1p description a bit. · 05b808ef
  Jared Casper authored Apr 05, 2023
  
  05b808ef
- A bit of cleanup. · 93bed794
  Jared Casper authored Apr 05, 2023
  
  93bed794
- Merge branch 'activations-bias' into 'main' · f6d36d03
  Jared Casper authored Apr 05, 2023
```
Add swiglu and squared relu activations and ability to disable bias.

See merge request ADLR/megatron-lm!553
```
  f6d36d03
- Merge branch 'bugFix' into 'main' · 22f054d8
  Shanmugam Ramasamy authored Apr 05, 2023
```
Update .gitlab-ci.yml

See merge request ADLR/megatron-lm!560
```
  22f054d8
05 Apr, 2023 4 commits
- Update .gitlab-ci.yml · 939383ae
  Shanmugam Ramasamy authored Apr 05, 2023
  
  939383ae
- Fix mpu init · 716da5d8
  MaximumEntropy authored Apr 05, 2023
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  716da5d8
- addressing the comments · 5ed304e4
  Mostofa Patwary authored Apr 05, 2023
  
  5ed304e4
- addressing the comments · 0760822b
  Mostofa Patwary authored Apr 05, 2023
  
  0760822b
04 Apr, 2023 4 commits
- Fix output layer init func when untie embeddings is True · 0f6bfdeb
  Sandeep Subramanian authored Feb 06, 2023
  
  0f6bfdeb
- Merge branch 'better-Output' into 'main' · 58020503
  Shanmugam Ramasamy authored Apr 04, 2023
```
Better output

See merge request ADLR/megatron-lm!559
```
  58020503
- Better output · 2221d882
  Shanmugam Ramasamy authored Apr 04, 2023
  
  2221d882
- removed self_attention_pos_emb · e1f1aa06
  Mostofa Patwary authored Apr 04, 2023
  
  e1f1aa06