Commits · 65eeb427db7f0bf66ca70d506ea09da7b0ae3032 · OpenDAS / Megatron-LM

26 Feb, 2021 1 commit
- Support Torch DDP for single-stage, num_microbatches() > 1 · 65eeb427
  Deepak Narayanan authored Feb 24, 2021
  
  65eeb427
25 Feb, 2021 3 commits
- Merge branch 'no-realm-import' into 'main' · 8cd16667
  Mohammad Shoeybi authored Feb 25, 2021
```
Don't import deprecated model from realm_model which is broken.

See merge request ADLR/megatron-lm!241
```
  8cd16667
- Don't import deprecated model from realm_model which is broken. · c6f18ccf
  Jared Casper authored Feb 25, 2021
  
  c6f18ccf
- Merge branch 'kernel_warning' into 'main' · 77d54fbb
  Jared Casper authored Feb 25, 2021
```
fix warning condition

See merge request ADLR/megatron-lm!238
```
  77d54fbb
23 Feb, 2021 2 commits
- fix warning condition · e055f3cf
  Vijay Korthikanti authored Feb 22, 2021
  
  e055f3cf
- Merge branch 'main_dedup' into 'main' · a1f4032b
  Jared Casper authored Feb 22, 2021
```
Storing and loading fingerprints of in deduplication

See merge request ADLR/megatron-lm!236
```
  a1f4032b
22 Feb, 2021 3 commits
- addressed the comments · ebc4a408
  Mostofa Patwary authored Feb 22, 2021
  
  ebc4a408
- Merge branch 'interleaved_schedule_assertion_fix' into 'main' · 90623142
  Jared Casper authored Feb 22, 2021
```
Fix interleaved schedule assertion

See merge request ADLR/megatron-lm!237
```
  90623142
- Fix interleaved schedule assertion · b7067cbd
  Deepak Narayanan authored Feb 22, 2021
  
  b7067cbd
19 Feb, 2021 4 commits
- updated the readme · 5a867963
  Mostofa Patwary authored Feb 19, 2021
  
  5a867963
- now multiple pickle files can be added · 4b821fb5
  Mostofa Patwary authored Feb 19, 2021
  
  4b821fb5
- Added an example · f6ddeca0
  Mostofa Patwary authored Feb 18, 2021
  
  f6ddeca0
- added storing and loading of pickle file of hash · 2f6e46bd
  Mostofa Patwary authored Feb 18, 2021
  
  2f6e46bd
18 Feb, 2021 9 commits
- Merge branch 'main_jaccard' into 'main' · 9dc111cc
  Jared Casper authored Feb 18, 2021
```
Bug Jaccard similarity and filtering n-grams

See merge request ADLR/megatron-lm!234
```
  9dc111cc
- Merge branch 'main_retriver_merge' into 'main' · dfd428cb
  Jared Casper authored Feb 18, 2021
```
ICT Retriever

See merge request ADLR/megatron-lm!235
```
  dfd428cb
- addressed the comments · 3f3ba5e7
  Mostofa Patwary authored Feb 18, 2021
  
  3f3ba5e7
- added readme · a449d312
  Mostofa Patwary authored Feb 18, 2021
  
  a449d312
- fixed an issue with ICT · 3d96729a
  Mostofa Patwary authored Feb 17, 2021
  
  3d96729a
- Merge branch 'main_retriver_merge' into 'main' · 68233932
  Jared Casper authored Feb 17, 2021
```
Merge ICT Retriever with Main branch

See merge request ADLR/megatron-lm!227
```
  68233932
- fixed a minor bug · 816fb890
  Mostofa Patwary authored Feb 17, 2021
  
  816fb890
- pulled latest megatron · b2f57fc4
  Mostofa Patwary authored Feb 17, 2021
  
  b2f57fc4
- Merge branch 'better_schedule' into 'main' · 76e3fca0
  Jared Casper authored Feb 17, 2021
```
Interleaved pipelined execution and refactored schedule / communication code

See merge request ADLR/megatron-lm!189
```
  76e3fca0
17 Feb, 2021 3 commits
- filtering n-grams · 59372322
  Mostofa Patwary authored Feb 17, 2021
  
  59372322
- Fixed a minor bug · 0a158128
  Mostofa Patwary authored Feb 17, 2021
  
  0a158128
- Turn scatter-gather optimization on by default · 72e4230b
  Deepak Narayanan authored Feb 16, 2021
  
  72e4230b
15 Feb, 2021 2 commits
- updated script as well · a4b628ab
  Mostofa Patwary authored Feb 15, 2021
  
  a4b628ab
- addressed the comments given by Mohammad · 447c1171
  Mostofa Patwary authored Feb 15, 2021
  
  447c1171
13 Feb, 2021 2 commits
- Change argument to control the number of model chunks in a stage · dcef9069
  Deepak Narayanan authored Feb 13, 2021
  
  dcef9069
- More comments and some cleanup (e.g., better variable names) · 5489bda9
  Deepak Narayanan authored Feb 13, 2021
  
  5489bda9
10 Feb, 2021 3 commits
- Comments in megatron/schedules.py and address a few more comments · 626645c0
  Deepak Narayanan authored Feb 10, 2021
  
  626645c0
- removed the function compile_helper · 22a3d81a
  Mostofa Patwary authored Feb 09, 2021
  
  22a3d81a
- Move unwrap to megatron/utils.py and clean up imports in megatron/schedules.py · cc691cbf
  Deepak Narayanan authored Feb 10, 2021
  
  cc691cbf
09 Feb, 2021 4 commits
- Compute tensor chunk size more cleanly, and add assertion for global batch size · e3e5ea89
  Deepak Narayanan authored Jan 20, 2021
  
  e3e5ea89
- Break up tensors sent between pipeline stages into smaller chunks that can be all-gathered · 27fc4689
  Deepak Narayanan authored Jan 20, 2021
  
  27fc4689
- Put in barriers in appropriate places to measure length of pipeline stall · 8e922d5b
  Deepak Narayanan authored Jan 09, 2021
  
  8e922d5b
- Interleaved pipeline execution and code refactoring · dd889062
  Deepak Narayanan authored Dec 12, 2020
```
- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions
```
  dd889062
08 Feb, 2021 2 commits
- Merge branch 'ckpt_rng' into 'main' · c1faa9fe
  Mohammad Shoeybi authored Feb 08, 2021
```
Improve handling of rng states in checkpoints.

See merge request ADLR/megatron-lm!231
```
  c1faa9fe
- Improve handling of rng states in checkpoints. · 08a848c7
  Jared Casper authored Feb 08, 2021
  
  08a848c7
06 Feb, 2021 1 commit

Merge branch 'main_fix' into 'main' · 8863af8c

Jared Casper authored Feb 05, 2021

Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API

See merge request ADLR/megatron-lm!230

8863af8c

05 Feb, 2021 1 commit
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 7ffea978
  Deepak Narayanan authored Feb 05, 2021
  
  7ffea978