Commits · 52a5f2f272e5ef242eb271227da712f7dfc55da3 · OpenDAS / Megatron-LM

12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

01 Oct, 2020 1 commit
- fix for nemo: do not initialize mpu if it is already initialized · 6e433055
  mohammad authored Oct 01, 2020
  
  6e433055
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
07 Aug, 2020 2 commits
- comment fix · 4e8ace9b
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  4e8ace9b
- fixed typo in comment · 39c80331
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  39c80331
06 Aug, 2020 3 commits
- Addressing code review comments · c632d205
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  c632d205
- Removed assert in case of external ddp, exported initialize_megatron() · be842037
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  be842037
- Implementing lazy parallel initialization · 5c04ceb3
  Boris Fomitchev authored Aug 05, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  5c04ceb3
30 Jul, 2020 1 commit
- Changes for NeMo/lightning compatibility · 417c7f6a
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  417c7f6a
22 Jul, 2020 1 commit
- Initialization fixes: allowing simple case like pytest pass, also making apex optional · 9026b86d
  Boris Fomitchev authored Jul 22, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  9026b86d
05 Jun, 2020 1 commit
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
21 May, 2020 1 commit
- Change sync variable to gloo backend · 05ea0cca
  Neel Kant authored May 21, 2020
  
  05ea0cca
19 May, 2020 1 commit
- Indexer_async works in theory · e338e311
  Neel Kant authored May 18, 2020
  
  e338e311
05 May, 2020 1 commit
- Update initialize.py · f931b8a3
  Raul Puri authored May 05, 2020
  
  f931b8a3
04 May, 2020 1 commit
- optionally allow no cpu · 6106127c
  Raul Puri authored May 03, 2020
  
  6106127c
21 Apr, 2020 1 commit
- added option to ignore unknown input arguments · 5531663b
  Mohammad authored Apr 20, 2020
  
  5531663b
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
11 Apr, 2020 2 commits
- Update initialize.py · 87bd026a
  Raul Puri authored Apr 10, 2020
  
  87bd026a
- Added logic for initilialize_megatron to work with cpu only jobs. This is... · 5448ca25
  Raul Puri authored Apr 10, 2020
```
Added logic for initilialize_megatron to work with cpu only jobs. This is necessary for several evaluation and processing scripts in downstream repos.
```
  5448ca25
01 Apr, 2020 1 commit
- addressed comments from raul, neel, and jared · bf3ce751
  Mohammad authored Apr 01, 2020
  
  bf3ce751
29 Mar, 2020 1 commit
- utils partially refactored and tested · 3f58649b
  Mohammad authored Mar 28, 2020
  
  3f58649b
28 Mar, 2020 2 commits
- arguments.py refactored · 94e2ca57
  Mohammad authored Mar 27, 2020
  
  94e2ca57
- added initialize megatron · a9e19f8e
  Mohammad authored Mar 27, 2020
  
  a9e19f8e