Commits · f5eac3d19b89991d2e2c88f50efeb0fe5cc69a45 · OpenDAS / Megatron-LM

07 Jan, 2021 1 commit
- Change some arguments to default to on. · 9acc8956
  Jared Casper authored Jan 06, 2021
  
  9acc8956
19 Dec, 2020 1 commit
- Fix TensorBoard writes · b81cad66
  mohammad authored Dec 12, 2020
  
  b81cad66
12 Nov, 2020 5 commits
- Move division of loss tensor by number of microbatches to training.py · c671de3e
  Deepak Narayanan authored Nov 12, 2020
  
  c671de3e
- Refactor code according to Jared's comments: move pipelining and... · 1979c242
  Deepak Narayanan authored Nov 12, 2020
```
Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size
```
  1979c242
- Divide gradient by number of microbatches in minibatch · 3d7194c4
  Deepak Narayanan authored Nov 03, 2020
  
  3d7194c4
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
07 Jul, 2020 1 commit
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
24 Jun, 2020 1 commit
- More ict_merge changes and interactive testing · 3354081f
  Neel Kant authored Jun 23, 2020
  
  3354081f
05 Jun, 2020 3 commits
- addressed jareds comments · 197c132e
  mohammad authored Jun 05, 2020
  
  197c132e
- added fp16 cross entropy loss option for gpt2 · acfe848e
  mohammad authored Jun 05, 2020
  
  acfe848e
- testing · 2ede8235
  mohammad authored Jun 05, 2020
  
  2ede8235
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
15 Apr, 2020 1 commit
- changed gpt2 masking to binary and masked_fill · 8ceed7c7
  Mohammad authored Apr 14, 2020
  
  8ceed7c7
13 Apr, 2020 1 commit
- data loading for BERT and GPT cleaned up · 2e38461b
  Mohammad authored Apr 12, 2020
  
  2e38461b
09 Apr, 2020 1 commit
- moved current gpt2 dataloader to old · 898fcb94
  Mohammad authored Apr 08, 2020
  
  898fcb94
03 Apr, 2020 1 commit
- generate samples linted · a6ba254f
  Mohammad authored Apr 02, 2020
  
  a6ba254f
02 Apr, 2020 1 commit
- refactored evaluate_gpt2 · 8fae1cdd
  Mohammad authored Apr 01, 2020
  
  8fae1cdd
01 Apr, 2020 1 commit
- addressed comments from raul, neel, and jared · bf3ce751
  Mohammad authored Apr 01, 2020
  
  bf3ce751
31 Mar, 2020 1 commit
- arguments in the model refactored · deffcb6a
  Mohammad authored Mar 31, 2020
  
  deffcb6a
30 Mar, 2020 3 commits
- both bert and gpt2 tested and working · 1788c910
  Mohammad authored Mar 29, 2020
  
  1788c910
- both bert and gpt are working · 5f8623db
  Mohammad authored Mar 29, 2020
  
  5f8623db
- added gpt2 tokenizer · 0e5dfd7f
  Mohammad authored Mar 29, 2020
  
  0e5dfd7f
29 Mar, 2020 2 commits
- refactored training · 27e14f82
  Mohammad authored Mar 28, 2020
  
  27e14f82
- utils partially refactored and tested · 3f58649b
  Mohammad authored Mar 28, 2020
  
  3f58649b
14 Jan, 2020 1 commit
- added query-key layer scaling and softmax fp32 option · 691747b1
  Mohammad Shoeybi authored Jan 13, 2020
  
  691747b1
27 Dec, 2019 1 commit
- checked and bert, gpt, and albert albert run · f86bb671
  Mohammad Shoeybi authored Dec 26, 2019
  
  f86bb671
26 Dec, 2019 1 commit
- fixed gpt-2 dataloder · d6485684
  Mohammad Shoeybi authored Dec 26, 2019
  
  d6485684
27 Nov, 2019 1 commit
- refactoring · f223ff67
  Mohammad Shoeybi authored Nov 27, 2019
  
  f223ff67
19 Nov, 2019 1 commit
- Plumbing for new albert dataset, including mods to arguments for data loaders. · 1a1cfcff
  Jared Casper authored Nov 18, 2019
  
  1a1cfcff
31 Oct, 2019 1 commit
- refacotred for code reuse · 34be7dd3
  Mohammad Shoeybi authored Oct 30, 2019
  
  34be7dd3
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290
09 Oct, 2019 1 commit
- moved padding to utils · 2e6d5ed9
  Mohammad Shoeybi authored Oct 09, 2019
  
  2e6d5ed9
08 Oct, 2019 2 commits
- moved few common elements between bert and gpt to utils · 787c1a0b
  Mohammad Shoeybi authored Oct 08, 2019
  
  787c1a0b
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
04 Oct, 2019 1 commit
- Updating public repo with latest changes. · a1d04b79
  Jared Casper authored Oct 04, 2019
  
  a1d04b79
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e