Commits · b2f57fc4b7305b281f3fc948c3461697a2d065e9 · OpenDAS / Megatron-LM

18 Feb, 2021 2 commits
- pulled latest megatron · b2f57fc4
  Mostofa Patwary authored Feb 17, 2021
  
  b2f57fc4
- Merge branch 'better_schedule' into 'main' · 76e3fca0
  Jared Casper authored Feb 17, 2021
```
Interleaved pipelined execution and refactored schedule / communication code

See merge request ADLR/megatron-lm!189
```
  76e3fca0
17 Feb, 2021 1 commit
- Turn scatter-gather optimization on by default · 72e4230b
  Deepak Narayanan authored Feb 16, 2021
  
  72e4230b
15 Feb, 2021 2 commits
- updated script as well · a4b628ab
  Mostofa Patwary authored Feb 15, 2021
  
  a4b628ab
- addressed the comments given by Mohammad · 447c1171
  Mostofa Patwary authored Feb 15, 2021
  
  447c1171
13 Feb, 2021 2 commits
- Change argument to control the number of model chunks in a stage · dcef9069
  Deepak Narayanan authored Feb 13, 2021
  
  dcef9069
- More comments and some cleanup (e.g., better variable names) · 5489bda9
  Deepak Narayanan authored Feb 13, 2021
  
  5489bda9
10 Feb, 2021 3 commits
- Comments in megatron/schedules.py and address a few more comments · 626645c0
  Deepak Narayanan authored Feb 10, 2021
  
  626645c0
- removed the function compile_helper · 22a3d81a
  Mostofa Patwary authored Feb 09, 2021
  
  22a3d81a
- Move unwrap to megatron/utils.py and clean up imports in megatron/schedules.py · cc691cbf
  Deepak Narayanan authored Feb 10, 2021
  
  cc691cbf
09 Feb, 2021 4 commits
- Compute tensor chunk size more cleanly, and add assertion for global batch size · e3e5ea89
  Deepak Narayanan authored Jan 20, 2021
  
  e3e5ea89
- Break up tensors sent between pipeline stages into smaller chunks that can be all-gathered · 27fc4689
  Deepak Narayanan authored Jan 20, 2021
  
  27fc4689
- Put in barriers in appropriate places to measure length of pipeline stall · 8e922d5b
  Deepak Narayanan authored Jan 09, 2021
  
  8e922d5b
- Interleaved pipeline execution and code refactoring · dd889062
  Deepak Narayanan authored Dec 12, 2020
```
- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions
```
  dd889062
08 Feb, 2021 2 commits
- Merge branch 'ckpt_rng' into 'main' · c1faa9fe
  Mohammad Shoeybi authored Feb 08, 2021
```
Improve handling of rng states in checkpoints.

See merge request ADLR/megatron-lm!231
```
  c1faa9fe
- Improve handling of rng states in checkpoints. · 08a848c7
  Jared Casper authored Feb 08, 2021
  
  08a848c7
06 Feb, 2021 1 commit

Merge branch 'main_fix' into 'main' · 8863af8c

Jared Casper authored Feb 05, 2021

Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API

See merge request ADLR/megatron-lm!230

8863af8c

05 Feb, 2021 3 commits
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 7ffea978
  Deepak Narayanan authored Feb 05, 2021
  
  7ffea978
- Merge branch 'fused_kernel_cond' into 'main' · 2096d356
  Jared Casper authored Feb 05, 2021
```
conditioning fused kernels

See merge request ADLR/megatron-lm!228
```
  2096d356
- address review comments · 0cb36de2
  Vijay Korthikanti authored Feb 05, 2021
  
  0cb36de2
04 Feb, 2021 5 commits
- conditioning fused kernels · 4916bae6
  Vijay Korthikanti authored Feb 04, 2021
  
  4916bae6
- updaded examples/pretrain_ict.sh · f7d96a69
  Mostofa Patwary authored Feb 03, 2021
  
  f7d96a69
- updaded examples/pretrain_ict.sh · 27f0f0ef
  Mostofa Patwary authored Feb 03, 2021
  
  27f0f0ef
- added reload_model_params function · 2cd5f18a
  Mostofa Patwary authored Feb 03, 2021
  
  2cd5f18a
- remove the function get_one_epoch_dataloader and also added assert · 8eff2a99
  Mostofa Patwary authored Feb 03, 2021
  
  8eff2a99
03 Feb, 2021 7 commits
- added examples/pretrain_ict.sh · 38898931
  Mostofa Patwary authored Feb 03, 2021
  
  38898931
- fixed arguments · 4e8f8c5e
  Mostofa Patwary authored Feb 03, 2021
  
  4e8f8c5e
- fixed arguments · 78a3dc32
  Mostofa Patwary authored Feb 03, 2021
  
  78a3dc32
- fixed exit interval · c380000e
  Mostofa Patwary authored Feb 03, 2021
  
  c380000e
- fixed alignment · 097fa65b
  Mostofa Patwary authored Feb 03, 2021
  
  097fa65b
- fixed alignment · a96ad63f
  Mostofa Patwary authored Feb 03, 2021
  
  a96ad63f
- Merging megatron with ICT · 651cc107
  Mostofa Patwary authored Feb 03, 2021
  
  651cc107
02 Feb, 2021 2 commits
- Merge branch 'merge_bugfix' into 'main' · 872e38ea
  Mohammad Shoeybi authored Feb 02, 2021
```
Fix bug in merge_mp_partitions for handling recent checkpoints.

See merge request ADLR/megatron-lm!226
```
  872e38ea
- Fix bug in merge_mp_partitions for handling recent checkpoints. · 72105ef0
  Jared Casper authored Feb 02, 2021
  
  72105ef0
01 Feb, 2021 4 commits
- Merge branch 'preprocess_fix' into 'main' · c601d751
  Mohammad Shoeybi authored Feb 01, 2021
```
Handle empty documents in preprocess_data.

See merge request ADLR/megatron-lm!225
```
  c601d751
- Handle empty documents in preprocess_data. · 09d220cf
  Jared Casper authored Feb 01, 2021
  
  09d220cf
- cleared the commented codes · e919dd8e
  Mostofa Patwary authored Feb 01, 2021
  
  e919dd8e
- fixed isse from Initializing ICT from pretrained BERT model · bfc20ecf
  Mostofa Patwary authored Feb 01, 2021
  
  bfc20ecf
29 Jan, 2021 2 commits
- Merge branch 'ci' into 'main' · 1b8e2891
  Jared Casper authored Jan 29, 2021
```
Init CI tests with very basic import test.

See merge request ADLR/megatron-lm!224
```
  1b8e2891
- Init CI tests with very basic import test. · 2526c614
  Jared Casper authored Jan 29, 2021
  
  2526c614