Commits · dfd8ed47c8b6d8faf26ee8fc2ad7dca2baf745f6 · OpenDAS / Megatron-LM

25 Dec, 2020 1 commit
- grads is removed from mpu · dfd8ed47
  mohammad authored Dec 25, 2020
  
  dfd8ed47
12 Nov, 2020 6 commits
- Clarifications in comments and minor refactoring to make main training loop more readable · 8fb2bc8c
  Deepak Narayanan authored Oct 28, 2020
  
  8fb2bc8c
- Simplify logic in megatron/fp16/fp16.py · 767e6e92
  Deepak Narayanan authored Oct 28, 2020
  
  767e6e92
- Small notes in comments in response to Jared's comments · aa9cae27
  Deepak Narayanan authored Oct 28, 2020
  
  aa9cae27
- Address Jared's comments in README and loss_scaler.py · dd079406
  Deepak Narayanan authored Oct 27, 2020
  
  dd079406
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
15 Apr, 2020 3 commits
- addressed jareds comment · a4489ffb
  Mohammad authored Apr 15, 2020
  
  a4489ffb
- addressed jareds comments · b6e73188
  Mohammad authored Apr 15, 2020
  
  b6e73188
- added multi-tensor-apply to fp16 · 99410264
  Mohammad authored Apr 14, 2020
  
  99410264
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb