Commits · e8fb052f810ab799ff947cbeb16f962ae32fc3b4 · OpenDAS / Megatron-LM

19 Aug, 2021 1 commit
- made contiguous buffer in local ddp default · e8fb052f
  mshoeybi authored Aug 18, 2021
  
  e8fb052f
18 Aug, 2021 2 commits
- set grads to none for the contig. buffers · df6e3cd7
  mshoeybi authored Aug 18, 2021
  
  df6e3cd7
- fixed flag name bug · db08a636
  Lawrence McAfee authored Aug 18, 2021
  
  db08a636
17 Aug, 2021 2 commits
- minor changes from github issues · 60078cf3
  mshoeybi authored Aug 17, 2021
  
  60078cf3
- added across rank sync for checkpoint iteration laod, fixed type for timing,... · 0722f878
  mshoeybi authored Aug 16, 2021
```
added across rank sync for checkpoint iteration laod, fixed type for timing, and validation iterations
```
  0722f878
16 Aug, 2021 2 commits
- added evaluation logic; finalized flag levels · 9dec5374
  Lawrence McAfee authored Aug 16, 2021
  
  9dec5374
- added flag/logic for emptying unused memory · 3bd2e973
  Lawrence McAfee authored Aug 16, 2021
  
  3bd2e973
21 Jul, 2021 1 commit
- Use helper method in megatron/schedules.py as intended · 77bff386
  Deepak Narayanan authored Jul 21, 2021
  
  77bff386
12 Jul, 2021 1 commit
- added memory stats (allocated/reserved) to tensorboard logging · 24b7c3c0
  Lawrence McAfee authored Jul 12, 2021
  
  24b7c3c0
03 Apr, 2021 1 commit
- Small bugfix to make sure refactored code works with interleaved schedule · 6fd78189
  Deepak Narayanan authored Apr 03, 2021
  
  6fd78189
02 Apr, 2021 1 commit
- Addressed MR comments, mostly adding comments to code. · e270f68a
  Jared Casper authored Apr 02, 2021
  
  e270f68a
24 Mar, 2021 1 commit
- pipeline code simplification · 3b91262e
  Vijay Korthikanti authored Mar 02, 2021
  
  3b91262e
19 Mar, 2021 1 commit
- Bfloat fused softmax + fused layer norm · 0fa7175f
  Mohammad Shoeybi authored Mar 19, 2021
  
  0fa7175f
08 Mar, 2021 1 commit
- Bfloat with fp32 grad acc · b4bc51b1
  Mohammad Shoeybi authored Mar 08, 2021
  
  b4bc51b1
04 Mar, 2021 1 commit
- Update argument names and fix merge error · 0aff3629
  Rewon Child authored Mar 04, 2021
  
  0aff3629
03 Mar, 2021 1 commit
- Remove pipeline stall timing (and avoid Explorer perf regressions) · ab09d819
  Deepak Narayanan authored Mar 03, 2021
  
  ab09d819
26 Feb, 2021 1 commit
- Support Torch DDP for single-stage, num_microbatches() > 1 · 65eeb427
  Deepak Narayanan authored Feb 24, 2021
  
  65eeb427
23 Feb, 2021 2 commits
- Add option to log zeros · 4e77e7c6
  Rewon Child authored Feb 23, 2021
  
  4e77e7c6
- Added code for building embeddings and savings · bcd605f8
  Mostofa Patwary authored Feb 23, 2021
  
  bcd605f8
22 Feb, 2021 1 commit
- Fix interleaved schedule assertion · b7067cbd
  Deepak Narayanan authored Feb 22, 2021
  
  b7067cbd
18 Feb, 2021 3 commits
- addressed the comments · 3f3ba5e7
  Mostofa Patwary authored Feb 18, 2021
  
  3f3ba5e7
- fixed an issue with ICT · 3d96729a
  Mostofa Patwary authored Feb 17, 2021
  
  3d96729a
- fixed a minor bug · 816fb890
  Mostofa Patwary authored Feb 17, 2021
  
  816fb890
13 Feb, 2021 1 commit
- More comments and some cleanup (e.g., better variable names) · 5489bda9
  Deepak Narayanan authored Feb 13, 2021
  
  5489bda9
10 Feb, 2021 2 commits
- Comments in megatron/schedules.py and address a few more comments · 626645c0
  Deepak Narayanan authored Feb 10, 2021
  
  626645c0
- Move unwrap to megatron/utils.py and clean up imports in megatron/schedules.py · cc691cbf
  Deepak Narayanan authored Feb 10, 2021
  
  cc691cbf
09 Feb, 2021 3 commits

Break up tensors sent between pipeline stages into smaller chunks that can be all-gathered · 27fc4689
Deepak Narayanan authored Jan 20, 2021

27fc4689
Put in barriers in appropriate places to measure length of pipeline stall · 8e922d5b
Deepak Narayanan authored Jan 09, 2021

8e922d5b

Interleaved pipeline execution and code refactoring · dd889062

Deepak Narayanan authored Dec 12, 2020

- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions

dd889062

06 Feb, 2021 1 commit
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 1b3dfa2f
  Deepak Narayanan authored Feb 05, 2021
  
  1b3dfa2f
05 Feb, 2021 1 commit
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 7ffea978
  Deepak Narayanan authored Feb 05, 2021
  
  7ffea978
04 Feb, 2021 1 commit
- added reload_model_params function · 2cd5f18a
  Mostofa Patwary authored Feb 03, 2021
  
  2cd5f18a
03 Feb, 2021 1 commit
- fixed exit interval · c380000e
  Mostofa Patwary authored Feb 03, 2021
  
  c380000e
01 Feb, 2021 2 commits
- cleared the commented codes · e919dd8e
  Mostofa Patwary authored Feb 01, 2021
  
  e919dd8e
- fixed isse from Initializing ICT from pretrained BERT model · bfc20ecf
  Mostofa Patwary authored Feb 01, 2021
  
  bfc20ecf
29 Jan, 2021 1 commit
- WIP: main_retriver_merge · 17d897e0
  Mostofa Patwary authored Jan 29, 2021
  
  17d897e0
28 Jan, 2021 2 commits
- changed validation loss name · 792a468d
  mohammad authored Jan 28, 2021
  
  792a468d
- added options for tensorboard logging · 3a26a168
  mohammad authored Jan 28, 2021
  
  3a26a168
27 Jan, 2021 1 commit
- added flag so we dont calculate params norm all the time · 3dcbaec9
  mohammad authored Jan 26, 2021
  
  3dcbaec9
26 Jan, 2021 1 commit
- added params norms · 57d1356e
  mohammad authored Jan 25, 2021
  
  57d1356e