Commits · 0024a5c66f90c7d3d02f7ef08a773aace6deb155 · OpenDAS / Megatron-LM

19 Apr, 2023 2 commits
- replace custom layer_norm_cuda with Apex layer_norm_cuda · d650e6a2
  Jon Barker authored Apr 19, 2023
  
  d650e6a2
- Some quick fixes to checkpoint_util. · be1a575e
  Jared Casper authored Apr 19, 2023
  
  be1a575e
07 Apr, 2023 1 commit
- Some quick fixes. · 45084d46
  Jared Casper authored Apr 06, 2023
  
  45084d46
06 Apr, 2023 5 commits
- dropout variable · a7cce320
  Jimmy Zhang authored Apr 06, 2023
  
  a7cce320
- refactor flash attention · 254e8815
  Jimmy Zhang authored Apr 06, 2023
  
  254e8815
- Flash Attention inference fix · f1a50a3c
  Jimmy Zhang authored Mar 03, 2023
  
  f1a50a3c
- Add assert in BertModel that untie_embeddings_and_output_weights is False · 1e2a0405
  Jared Casper authored Apr 05, 2023
  
  1e2a0405
- A bit of cleanup. · 93bed794
  Jared Casper authored Apr 05, 2023
  
  93bed794
05 Apr, 2023 2 commits
- addressing the comments · 5ed304e4
  Mostofa Patwary authored Apr 05, 2023
  
  5ed304e4
- addressing the comments · 0760822b
  Mostofa Patwary authored Apr 05, 2023
  
  0760822b
04 Apr, 2023 3 commits
- Fix output layer init func when untie embeddings is True · 0f6bfdeb
  Sandeep Subramanian authored Feb 06, 2023
  
  0f6bfdeb
- removed self_attention_pos_emb · e1f1aa06
  Mostofa Patwary authored Apr 04, 2023
  
  e1f1aa06
- addressing comments · bf1da27e
  Mostofa Patwary authored Apr 04, 2023
  
  bf1da27e
03 Apr, 2023 10 commits
- addressing more comments · 5b7736ca
  Mostofa Patwary authored Apr 03, 2023
  
  5b7736ca
- addressing more comments · 07ffa1b7
  Mostofa Patwary authored Apr 03, 2023
  
  07ffa1b7
- remove unneccessary argument check · a9f6569d
  Jimmy Zhang authored Apr 03, 2023
  
  a9f6569d
- assertion check for T5 and untied embeddings · 6b50a8c6
  Jimmy Zhang authored Apr 03, 2023
  
  6b50a8c6
- addressed the comments · df521589
  Mostofa Patwary authored Apr 03, 2023
  
  df521589
- Save word embeddings for head only only if embeddings are not untied · 18b26ec6
  Sandeep Subramanian authored Feb 15, 2023
  
  18b26ec6
- disable embedding addreduce if untie_embeddings_and_output_weights · f11b4c99
  Jimmy Zhang authored Jan 31, 2023
  
  f11b4c99
- Fix checkpointing · 36c3a1c1
  MaximumEntropy authored Dec 08, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  36c3a1c1
- Initial commit for untied embeddings · 82c7ba57
  MaximumEntropy authored Dec 07, 2022
```
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
```
  82c7ba57
- Addressing comments. · 420eec74
  Jared Casper authored Apr 03, 2023
  
  420eec74
02 Apr, 2023 1 commit
- layernorm1p added · 5e079c87
  Mostofa Patwary authored Apr 02, 2023
  
  5e079c87
01 Apr, 2023 1 commit
- Add swiglu and squared relu activations and ability to disable bias. · 2cc3dac7
  Jared Casper authored Apr 01, 2023
  
  2cc3dac7
31 Mar, 2023 2 commits
- merging rope to main · 1e0e555c
  Mostofa Patwary authored Mar 31, 2023
  
  1e0e555c
- fix bug in uniform activation recompute · fc2c81d3
  Jared Casper authored Mar 31, 2023
  
  fc2c81d3
23 Mar, 2023 1 commit
- Move pipeline parallel functionality into core with associated changes. · 3c92fa93
  Jared Casper authored Mar 23, 2023
  
  3c92fa93
22 Feb, 2023 1 commit
- Retro · 17a6044d
  Lawrence McAfee authored Feb 21, 2023
  
  17a6044d
04 Jan, 2023 1 commit
- Remove FA's check for headdim <= 128 · 9200e43a
  Tri Dao authored Jan 03, 2023
  
  9200e43a
22 Dec, 2022 1 commit
- Transformer Engine Integration Rebase · 3499542e
  John Kamalu authored Dec 21, 2022
  
  3499542e
09 Dec, 2022 1 commit
- Integrate FlashAttention into Megatron-LM · d693034e
  Tri Dao authored Dec 08, 2022
  
  d693034e
22 Nov, 2022 1 commit
- Fix DropPath for hidden shape [s, b, h] · 6d45a903
  Yu Yao authored Nov 22, 2022
  
  6d45a903
10 Nov, 2022 1 commit
- ViT Backbone Tensor Shape Fix · fabd3e4e
  Yu Yao authored Nov 10, 2022
  
  fabd3e4e
08 Nov, 2022 1 commit
- Fix merge error. · e0a12fe1
  Jared Casper authored Nov 08, 2022
  
  e0a12fe1
04 Oct, 2022 1 commit
- different encoder/decoder num-layers support · cbf8250b
  Vijay Korthikanti authored Oct 03, 2022
  
  cbf8250b
24 Sep, 2022 2 commits
- Move get_num_layers into transformer.py. · a94d0a6e
  Jared Casper authored Sep 23, 2022
  
  a94d0a6e
- Alias core.parallel_state as mpu and use it throughout code. RIP mpu. · 5942af97
  Jared Casper authored Sep 23, 2022
  
  5942af97
23 Sep, 2022 2 commits
- Move layers from mpu to core.tensor_parallel. · c2ea914f
  Jared Casper authored Sep 23, 2022
  
  c2ea914f
- Start Megatron-Core with vocab parallel cross entropy · 2e6a46e4
  Jared Casper authored Sep 22, 2022
```
Also merged in some changed from apex
```
  2e6a46e4