Commits · adebe364f0d714bc7ef6da94f3b70ab37c3d7c2b · OpenDAS / Megatron-LM

28 Jan, 2022 5 commits
- Merge branch 'patch-1' of https://github.com/jamesr66a/Megatron-LM into github-pr · adebe364
  Jared Casper authored Jan 28, 2022
  
  adebe364
- Merge branch 'patch-2' of https://github.com/kvtoraman/Megatron-LM into github-pr · 20f6169f
  Jared Casper authored Jan 28, 2022
  
  20f6169f
- Merge branch 'patch-1' of https://github.com/rajeshkppt/Megatron-LM into github-pr · 0747e8e5
  Jared Casper authored Jan 28, 2022
  
  0747e8e5
- Merge branch 'fix' of https://github.com/singleheart/Megatron-LM into github-pr · 9882fb3f
  Jared Casper authored Jan 28, 2022
  
  9882fb3f
- Merge branch 'patch-1' of https://github.com/stas00/Megatron-LM into github-pr · 4a62d582
  Jared Casper authored Jan 28, 2022
  
  4a62d582
27 Jan, 2022 3 commits
- Merge branch 'lmcafee/distrib-chkpt-fix-v2' into 'main' · fd5469aa
  Jared Casper authored Jan 26, 2022
```
Distributed checkpointing memory fix

See merge request ADLR/megatron-lm!379
```
  fd5469aa
- add clarification about the model parallel size column · 2d767f48
  Stas Bekman authored Jan 26, 2022
  
  2d767f48
- [README] specify explicitly which gpu and node size was used · 145d2eb9
  Stas Bekman authored Jan 26, 2022
```
The paper has this info, so proposing to copy it next to the table. 

Otherwise it's hard to guess whether you used 40GB A100s or 80GB ones (and secondary, n_gpus per node).

Thank you!
```
  145d2eb9
26 Jan, 2022 1 commit
- further clarified viewless tensor comment in transformer.py · d16e2a24
  Lawrence McAfee authored Jan 26, 2022
  
  d16e2a24
25 Jan, 2022 2 commits
- Merge branch 'lmcafee/t5-free-pipe-output-true-fix' into 'main' · fc5d4c2b
  Jared Casper authored Jan 25, 2022
```
T5 fix for deallocating pipeline output tensors

See merge request ADLR/megatron-lm!380
```
  fc5d4c2b
- limit 'make_viewless_tensor()' to case of micro_batch_size == 1; added comment · 24369dd6
  Lawrence McAfee authored Jan 25, 2022
  
  24369dd6
22 Jan, 2022 1 commit
- fix an argument · 313730e9
  Jay authored Jan 22, 2022
  
  313730e9
20 Jan, 2022 2 commits
- Merge branch 'vision-merge' into 'main' · 539e39b6
  Jared Casper authored Jan 20, 2022
```
Phase1 merge: vit optimizations + dataset enhancements + scaled_softmax kernel

See merge request ADLR/megatron-lm!378
```
  539e39b6
- address review comments · 798b6a64
  Vijay Korthikanti authored Jan 20, 2022
  
  798b6a64
19 Jan, 2022 2 commits
- using all_gather instead of gather (nccl does not support gather) · b037a69e
  Vijay Korthikanti authored Jan 19, 2022
  
  b037a69e
- flag for data parallel random initialization · a7ee77ea
  Vijay Korthikanti authored Jan 19, 2022
  
  a7ee77ea
15 Jan, 2022 2 commits
- minor fix · fd8dd9c0
  Vijay Korthikanti authored Jan 14, 2022
  
  fd8dd9c0
- checkpointing rng_state of all data parallel ranks · 09d38884
  Vijay Korthikanti authored Jan 14, 2022
  
  09d38884
14 Jan, 2022 1 commit
- address review comments · 343dc97a
  Vijay Korthikanti authored Jan 14, 2022
  
  343dc97a
13 Jan, 2022 7 commits
- handle output_tensor == None · 7f1c5917
  Lawrence McAfee authored Jan 13, 2022
  
  7f1c5917
- more cleanup · 8fc5e323
  Lawrence McAfee authored Jan 13, 2022
  
  8fc5e323
- renamed 'free' -> 'deallocate' · c1e4526b
  Lawrence McAfee authored Jan 13, 2022
  
  c1e4526b
- removed uses are args.deallocate_pipeline_output · d10f81c5
  Lawrence McAfee authored Jan 13, 2022
  
  d10f81c5
- removed argument flag; also, converted torch.FloatTenstor -> torch.empty in previous commit · 18846a0a
  Lawrence McAfee authored Jan 13, 2022
  
  18846a0a
- fixed bug for no-interleave pipeline schedule · 1f387c2c
  Lawrence McAfee authored Jan 13, 2022
  
  1f387c2c
- minor fixes · 3b29f0e7
  Vijay Korthikanti authored Jan 13, 2022
  
  3b29f0e7
12 Jan, 2022 1 commit
- Phase1 merge: vit optimizations + dataset enhancements + scaled_softmax kernel · 7a77abd9
  Vijay Korthikanti authored Jan 12, 2022
  
  7a77abd9
11 Jan, 2022 4 commits
- removed several 'assert_viewless_tensor' calls. · 0c8e8cce
  Lawrence McAfee authored Jan 11, 2022
  
  0c8e8cce
- added comments · a1fe4805
  Lawrence McAfee authored Jan 11, 2022
  
  a1fe4805
- partially cleaned · 806422e5
  Lawrence McAfee authored Jan 11, 2022
  
  806422e5
- jan 11 alpha · 05042081
  Lawrence McAfee authored Jan 11, 2022
  
  05042081
10 Jan, 2022 1 commit
- loss matches; memory savings for multi-node (tested n3, n16) · 270d6412
  Lawrence McAfee authored Jan 10, 2022
  
  270d6412
08 Jan, 2022 2 commits
- jan 7 beta · b6d4369b
  Lawrence McAfee authored Jan 07, 2022
  
  b6d4369b
- more iterating on 'viewless tensor' methods · ed0c8714
  Lawrence McAfee authored Jan 07, 2022
  
  ed0c8714
07 Jan, 2022 1 commit
- debugging make_standalone_tensor(), safely_set_tensor_data_attr() · 5422d23a
  Lawrence McAfee authored Jan 07, 2022
  
  5422d23a
05 Jan, 2022 1 commit
- Merge branch 'free-pipe-output-t5-fix' into 'main' · 9a8b89ac
  Jared Casper authored Jan 04, 2022
```
added flag to control deallocation of pipeline outputs

See merge request ADLR/megatron-lm!377
```
  9a8b89ac
04 Jan, 2022 3 commits
- added flag to control deallocation of pipeline outputs · de1ae5b2
  Lawrence McAfee authored Jan 04, 2022
  
  de1ae5b2
- Merge branch 't5_pipeline_fixes' into 'main' · 2de7ae27
  Jared Casper authored Jan 04, 2022
```
Generic fix to T5 pipeline parallelism bug.

See merge request ADLR/megatron-lm!376
```
  2de7ae27
- update comment · d439b417
  Vijay Korthikanti authored Jan 04, 2022
  
  d439b417
31 Dec, 2021 1 commit
- simplify code · 9da6e975
  Vijay Korthikanti authored Dec 30, 2021
  
  9da6e975