Commits · feecd5d9d083365deacdcf87d40c11e265f6bea7 · OpenDAS / Megatron-LM

19 Dec, 2020 2 commits
- Add constant num micro-batches calculator · feecd5d9
  mohammad authored Dec 07, 2020
  
  feecd5d9
- Rename --batch-size to --micro-batch-size and drop in-minibatch from... · 9019bbf4
  mohammad authored Dec 06, 2020
```
Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch
```
  9019bbf4
12 Nov, 2020 3 commits
- Back compatibility of checkpoints: don't rename model_parallel_rng_tracker · e805f0bd
  Deepak Narayanan authored Oct 22, 2020
  
  e805f0bd
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
07 Aug, 2020 1 commit
- Fix typo · f63a8a12
  Nako Sung authored Aug 08, 2020
```
Tiny typo fix : sedd -> seed
```
  f63a8a12
04 Jul, 2020 1 commit
- small fix for CheckpointFunction's backward() method when sone args may be NoneType · 4ee0537a
  Devendra Singh Sachan authored Jul 04, 2020
  
  4ee0537a
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
12 Sep, 2019 1 commit
- Support latest PyTorch RNG state API. (#8) · 2ef23675
  Jared Casper authored Sep 11, 2019
```
Fixes #7.
```
  2ef23675
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e