Commits · 7abd3e90d09cf02a3a019c9a2af67ee31d5b1bd4 · OpenDAS / Megatron-LM

12 Nov, 2020 1 commit

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

02 Oct, 2020 1 commit
- Fixed uninit communication gruop · 057a1fab
  Mostofa Patwary authored Oct 01, 2020
  
  057a1fab
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
03 Sep, 2020 1 commit
- debuged the cpu init for mp=1 · 04cd82bd
  mohammad authored Sep 03, 2020
  
  04cd82bd
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
26 Aug, 2020 1 commit
- Making --lazy_mpu_init act as _USE_CPU_INITIALIZATION · e10760ea
  Boris Fomitchev authored Aug 26, 2020
  
  e10760ea
06 Aug, 2020 1 commit
- Implementing lazy parallel initialization · 5c04ceb3
  Boris Fomitchev authored Aug 05, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  5c04ceb3
03 Aug, 2020 1 commit
- added gpu initialization and option to avoid master values · 0403b808
  mohammad authored Aug 03, 2020
  
  0403b808
31 Jul, 2020 1 commit
- Code review comments - changing parallel test condition · 35bea728
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  35bea728
30 Jul, 2020 1 commit
- Changes for NeMo/lightning compatibility · 417c7f6a
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  417c7f6a
26 Jul, 2020 2 commits
- Add additional assertion on Indexer to test correctness, and limit verbosity in other classes · eaa5d877
  Neel Kant authored Jul 25, 2020
  
  eaa5d877
- Complete fix for APEX absence in NeMo · aa0ee72e
  Boris Fomitchev authored Jul 25, 2020
  
  aa0ee72e
24 Jul, 2020 1 commit
- ONNX export fix, including one for gelu() · 158a99c3
  Boris Fomitchev authored Jul 23, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  158a99c3
04 Jul, 2020 1 commit
- small fix for CheckpointFunction's backward() method when sone args may be NoneType · 4ee0537a
  Devendra Singh Sachan authored Jul 04, 2020
  
  4ee0537a
05 Jun, 2020 1 commit
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
26 May, 2020 1 commit
- Corrected realm example building, misc improvements for async concurrency · 2fd4ea6c
  Neel Kant authored May 25, 2020
  
  2fd4ea6c
21 May, 2020 1 commit
- Change sync variable to gloo backend · 05ea0cca
  Neel Kant authored May 21, 2020
  
  05ea0cca
20 May, 2020 1 commit
- Async works for total 8 GPU, indexer debug mode · a670b6c9
  Neel Kant authored May 19, 2020
  
  a670b6c9
19 May, 2020 2 commits
- Reorganize indexer. Things run up to saving model checkpoint and repeating · d4b00be0
  Neel Kant authored May 19, 2020
  
  d4b00be0
- Indexer_async works in theory · e338e311
  Neel Kant authored May 18, 2020
  
  e338e311
21 Apr, 2020 3 commits
- removed optimized l2 grad clipping · 0b6bdca6
  Mohammad authored Apr 21, 2020
  
  0b6bdca6
- fixed a bug in l2 grad clip · 4ff2c963
  Mohammad authored Apr 20, 2020
  
  4ff2c963
- fixed a bug in l2 grad clip · fb5b2b36
  Mohammad authored Apr 20, 2020
  
  fb5b2b36
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
15 Apr, 2020 1 commit
- added faster L2 grad clipping and new torch gelu · 3ea5491e
  Mohammad authored Apr 14, 2020
  
  3ea5491e
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
26 Mar, 2020 1 commit
- Modify pretrain_bert_ict.py to work with ICTBertModel · b1efc33d
  Neel Kant authored Mar 26, 2020
  
  b1efc33d
11 Mar, 2020 1 commit
- memory optimization in mpu cross entropy · 57064fd6
  Mohammad Shoeybi authored Mar 10, 2020
  
  57064fd6
10 Feb, 2020 1 commit
- Model parallel merger · 57c2060f
  Mohammad Shoeybi authored Feb 10, 2020
  
  57c2060f
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb