Commits · 9b174da873cf8e84f1bef1b6a88d98ff7791946e · OpenDAS / Megatron-LM

19 Dec, 2020 2 commits
- Better memory tracking across pipeline-parallel ranks · 3574b8e6
  Deepak Narayanan authored Dec 06, 2020
  
  3574b8e6
- Rename --batch-size to --micro-batch-size and drop in-minibatch from... · 9019bbf4
  mohammad authored Dec 06, 2020
```
Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch
```
  9019bbf4
28 Nov, 2020 1 commit
- added consumed tokens to checkpoints and some refactoring · f0a445fa
  mohammad authored Nov 27, 2020
  
  f0a445fa
26 Nov, 2020 1 commit
- simplified sampler · 4311b695
  mohammad authored Nov 25, 2020
  
  4311b695
18 Nov, 2020 1 commit
- Replace deprecated torch.norm with torch.linalg.norm. · 17035d6c
  Jared Casper authored Nov 17, 2020
  
  17035d6c
12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
07 Jul, 2020 1 commit
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
24 Jun, 2020 1 commit
- More ict_merge changes and interactive testing · 3354081f
  Neel Kant authored Jun 23, 2020
  
  3354081f
05 Jun, 2020 1 commit
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
24 May, 2020 1 commit
- Use Faiss GPU index and report retrieval utility · 8573ab35
  Neel Kant authored May 24, 2020
  
  8573ab35
20 May, 2020 1 commit
- Async works for total 8 GPU, indexer debug mode · a670b6c9
  Neel Kant authored May 19, 2020
  
  a670b6c9
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
15 Apr, 2020 1 commit
- changed gpt2 masking to binary and masked_fill · 8ceed7c7
  Mohammad authored Apr 14, 2020
  
  8ceed7c7
14 Apr, 2020 2 commits
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
- Re-adjust dataloader properties · 064a6881
  Neel Kant authored Apr 13, 2020
  
  064a6881
07 Apr, 2020 1 commit
- Fix issue with validation dataloader · 064bdc46
  Neel Kant authored Apr 07, 2020
  
  064bdc46
02 Apr, 2020 1 commit
- refactored evaluate_gpt2 · 8fae1cdd
  Mohammad authored Apr 01, 2020
  
  8fae1cdd
30 Mar, 2020 2 commits
- added samplers from data_utils · 32e1ddb0
  Mohammad authored Mar 30, 2020
  
  32e1ddb0
- both bert and gpt2 tested and working · 1788c910
  Mohammad authored Mar 29, 2020
  
  1788c910
29 Mar, 2020 3 commits
- refactored learning-rate · b6e0377b
  Mohammad authored Mar 29, 2020
  
  b6e0377b
- utils partially refactored and tested · 3f58649b
  Mohammad authored Mar 28, 2020
  
  3f58649b
- refactored checkpooints but not tested yet · 86e7d624
  Mohammad authored Mar 28, 2020
  
  86e7d624
28 Mar, 2020 1 commit
- working on utils · 5050203f
  Mohammad authored Mar 28, 2020
  
  5050203f
27 Mar, 2020 1 commit
- added global variables · 83aa9219
  Mohammad authored Mar 27, 2020
  
  83aa9219
17 Jan, 2020 1 commit
- undid padding merge · 7a6d630e
  Raul Puri authored Jan 16, 2020
  
  7a6d630e
22 Dec, 2019 1 commit
- force output gathering · b9b6fe0d
  Raul Puri authored Dec 22, 2019
  
  b9b6fe0d
27 Nov, 2019 1 commit
- refactoring · f223ff67
  Mohammad Shoeybi authored Nov 27, 2019
  
  f223ff67
31 Oct, 2019 1 commit
- refacotred for code reuse · 34be7dd3
  Mohammad Shoeybi authored Oct 30, 2019
  
  34be7dd3
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290
09 Oct, 2019 2 commits
- moved padding to utils · 2e6d5ed9
  Mohammad Shoeybi authored Oct 09, 2019
  
  2e6d5ed9
- fixed deserializing issue with old checkpoint · ee38e7f9
  Mohammad Shoeybi authored Oct 09, 2019
  
  ee38e7f9
08 Oct, 2019 2 commits
- moved few common elements between bert and gpt to utils · 787c1a0b
  Mohammad Shoeybi authored Oct 08, 2019
  
  787c1a0b
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
04 Oct, 2019 1 commit
- Updating public repo with latest changes. · a1d04b79
  Jared Casper authored Oct 04, 2019
  
  a1d04b79
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e

14 May, 2019 1 commit
- fixed save race condition · 0399d32c
  Raul Puri authored May 14, 2019
  
  0399d32c
27 Mar, 2019 1 commit
- initial commit · fb4cbdc2
  Raul Puri authored Mar 27, 2019
  
  fb4cbdc2