Commits · fdcaeba044fcdfe8a3e247d7f842c475a6171f55 · OpenDAS / Megatron-LM

12 Nov, 2020 17 commits
- Remove timing labels that don't make sense · fdcaeba0
  Deepak Narayanan authored Nov 10, 2020
  
  fdcaeba0
- Better communication timing · 7d367b1d
  Deepak Narayanan authored Nov 10, 2020
  
  7d367b1d
- Divide gradient by number of microbatches in minibatch · 3d7194c4
  Deepak Narayanan authored Nov 03, 2020
  
  3d7194c4
- Better 'forward' and 'backward' timing in megatron/training.py · a6756bf8
  Deepak Narayanan authored Nov 03, 2020
  
  a6756bf8
- Move training schedule to 1F1B for memory efficiency · 3e6898e6
  Deepak Narayanan authored Oct 24, 2020
  
  3e6898e6
- Only transpose hidden_states when necessary · 6abf39be
  Deepak Narayanan authored Nov 03, 2020
  
  6abf39be
- Log times for various sub-operations in forward and backward pass in main training loop · eed0062a
  Deepak Narayanan authored Oct 30, 2020
  
  eed0062a
- Bugfix in megatron/training.py: correct global_batch_size computation · 7ce373f3
  Deepak Narayanan authored Oct 29, 2020
```
Prevents data_loader from running out of training examples
```
  7ce373f3
- Improve names of identifiers used for timing in main training loop · 9d4c735a
  Deepak Narayanan authored Oct 29, 2020
  
  9d4c735a
- Clarifications in comments and minor refactoring to make main training loop more readable · 8fb2bc8c
  Deepak Narayanan authored Oct 28, 2020
  
  8fb2bc8c
- Bugfix in main training loop: Update master_grads only after grads are correctly accumulated · 9b558566
  Deepak Narayanan authored Oct 28, 2020
  
  9b558566
- Small notes in comments in response to Jared's comments · aa9cae27
  Deepak Narayanan authored Oct 28, 2020
  
  aa9cae27
- Improve time logging when num_microbatches_in_minibatch > 1 · 63740223
  Deepak Narayanan authored Oct 27, 2020
```
Make sure all forward and backward operations are accounted for
```
  63740223
- Refactor communication code in main training loop to helper method · 318d68c2
  Deepak Narayanan authored Oct 23, 2020
  
  318d68c2
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
- fp32 working · 28cd66e1
  mohammad authored Aug 28, 2020
  
  28cd66e1
03 Nov, 2020 1 commit
- fixed loss average when all but one value is skipped · 664cd28b
  mohammad authored Nov 03, 2020
  
  664cd28b
14 Oct, 2020 1 commit
- added adam betas and eps as arguments · 48269d8d
  Mostofa Patwary authored Oct 13, 2020
  
  48269d8d
25 Sep, 2020 1 commit
- fixed the interation count · 2db01991
  mohammad authored Sep 24, 2020
  
  2db01991
24 Sep, 2020 1 commit
- nan values report added · abd7636e
  mohammad authored Sep 23, 2020
  
  abd7636e
22 Sep, 2020 1 commit
- fixed logging bug · a74f8d9c
  mohammad authored Sep 22, 2020
  
  a74f8d9c
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
14 Jul, 2020 1 commit
- Additional refactoring · b9bd1a11
  Neel Kant authored Jul 13, 2020
  
  b9bd1a11
07 Jul, 2020 1 commit
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
25 Jun, 2020 1 commit
- Add stanza NER salient span masking · e7045139
  Neel Kant authored Jun 24, 2020
  
  e7045139
24 Jun, 2020 1 commit
- More ict_merge changes and interactive testing · 3354081f
  Neel Kant authored Jun 23, 2020
  
  3354081f
22 Jun, 2020 1 commit
- Address most of comments from code review w/ Mohammad · 6495927e
  Neel Kant authored Jun 22, 2020
  
  6495927e
15 Jun, 2020 1 commit
- Debugging done on Circe · b8bb0b49
  Mohammad authored Jun 15, 2020
  
  b8bb0b49
09 Jun, 2020 1 commit
- Correct retrieval utility and add salient span preprocessing · bf599e86
  Neel Kant authored Jun 08, 2020
  
  bf599e86
05 Jun, 2020 4 commits
- added fp16 cross entropy loss option for gpt2 · acfe848e
  mohammad authored Jun 05, 2020
  
  acfe848e
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
- runs with new log loss but plateaus early · 20895f2c
  Neel Kant authored Jun 04, 2020
  
  20895f2c
- testing · 2ede8235
  mohammad authored Jun 05, 2020
  
  2ede8235
31 May, 2020 1 commit
- Misc changes · 51204a4d
  Neel Kant authored May 30, 2020
  
  51204a4d
27 May, 2020 2 commits
- Correct indexer seq length · dfb907fe
  Neel Kant authored May 27, 2020
  
  dfb907fe
- Correct CrossEntropyLoss · 0e8f4331
  Neel Kant authored May 27, 2020
  
  0e8f4331
26 May, 2020 1 commit
- Corrected realm example building, misc improvements for async concurrency · 2fd4ea6c
  Neel Kant authored May 25, 2020
  
  2fd4ea6c
24 May, 2020 1 commit
- Fix token alignment, add mpu checkpointing, misc training code · 8e22824e
  Neel Kant authored May 24, 2020
  
  8e22824e
21 May, 2020 1 commit
- Change sync variable to gloo backend · 05ea0cca
  Neel Kant authored May 21, 2020
  
  05ea0cca