Commits · bcd605f8570ebeeb0436c115ebbfafc3c5a40ae5 · OpenDAS / Megatron-LM

23 Feb, 2021 1 commit
- Added code for building embeddings and savings · bcd605f8
  Mostofa Patwary authored Feb 23, 2021
  
  bcd605f8
19 Feb, 2021 1 commit
- Merge branch 'main_retriver_merge' into main_retriver_merge_ict_eval · 31d39ec0
  Mostofa Patwary authored Feb 19, 2021
  
  31d39ec0
18 Feb, 2021 10 commits
- Merge branch 'main_jaccard' into 'main' · 9dc111cc
  Jared Casper authored Feb 18, 2021
```
Bug Jaccard similarity and filtering n-grams

See merge request ADLR/megatron-lm!234
```
  9dc111cc
- Merge branch 'main_retriver_merge' into 'main' · dfd428cb
  Jared Casper authored Feb 18, 2021
```
ICT Retriever

See merge request ADLR/megatron-lm!235
```
  dfd428cb
- addressed the comments · 3f3ba5e7
  Mostofa Patwary authored Feb 18, 2021
  
  3f3ba5e7
- added readme · a449d312
  Mostofa Patwary authored Feb 18, 2021
  
  a449d312
- fixed an issue with ICT · 3d96729a
  Mostofa Patwary authored Feb 17, 2021
  
  3d96729a
- Merge branch 'main_retriver_merge' into 'main' · 68233932
  Jared Casper authored Feb 17, 2021
```
Merge ICT Retriever with Main branch

See merge request ADLR/megatron-lm!227
```
  68233932
- fixed a minor bug · 816fb890
  Mostofa Patwary authored Feb 17, 2021
  
  816fb890
- pulled latest megatron · b2f57fc4
  Mostofa Patwary authored Feb 17, 2021
  
  b2f57fc4
- evaluation code ongoing · 612f438a
  Mostofa Patwary authored Feb 17, 2021
  
  612f438a
- Merge branch 'better_schedule' into 'main' · 76e3fca0
  Jared Casper authored Feb 17, 2021
```
Interleaved pipelined execution and refactored schedule / communication code

See merge request ADLR/megatron-lm!189
```
  76e3fca0
17 Feb, 2021 4 commits
- Merge branch 'main_retriver_merge' into main_retriver_merge_ict_eval · ebc95c35
  Mostofa Patwary authored Feb 17, 2021
  
  ebc95c35
- filtering n-grams · 59372322
  Mostofa Patwary authored Feb 17, 2021
  
  59372322
- Fixed a minor bug · 0a158128
  Mostofa Patwary authored Feb 17, 2021
  
  0a158128
- Turn scatter-gather optimization on by default · 72e4230b
  Deepak Narayanan authored Feb 16, 2021
  
  72e4230b
15 Feb, 2021 3 commits
- updated script as well · a4b628ab
  Mostofa Patwary authored Feb 15, 2021
  
  a4b628ab
- addressed the comments given by Mohammad · 447c1171
  Mostofa Patwary authored Feb 15, 2021
  
  447c1171
- minor change · 360fb6b7
  Mostofa Patwary authored Feb 15, 2021
  
  360fb6b7
13 Feb, 2021 2 commits
- Change argument to control the number of model chunks in a stage · dcef9069
  Deepak Narayanan authored Feb 13, 2021
  
  dcef9069
- More comments and some cleanup (e.g., better variable names) · 5489bda9
  Deepak Narayanan authored Feb 13, 2021
  
  5489bda9
11 Feb, 2021 1 commit
- Merge branch 'main_retriver_merge' into main_retriver_merge_ict_eval · 4bea4e76
  Mostofa Patwary authored Feb 11, 2021
  
  4bea4e76
10 Feb, 2021 4 commits
- Comments in megatron/schedules.py and address a few more comments · 626645c0
  Deepak Narayanan authored Feb 10, 2021
  
  626645c0
- removed the function compile_helper · 22a3d81a
  Mostofa Patwary authored Feb 09, 2021
  
  22a3d81a
- started evaluation of ICT · 613d0fe8
  Mostofa Patwary authored Feb 09, 2021
  
  613d0fe8
- Move unwrap to megatron/utils.py and clean up imports in megatron/schedules.py · cc691cbf
  Deepak Narayanan authored Feb 10, 2021
  
  cc691cbf
09 Feb, 2021 4 commits
- Compute tensor chunk size more cleanly, and add assertion for global batch size · e3e5ea89
  Deepak Narayanan authored Jan 20, 2021
  
  e3e5ea89
- Break up tensors sent between pipeline stages into smaller chunks that can be all-gathered · 27fc4689
  Deepak Narayanan authored Jan 20, 2021
  
  27fc4689
- Put in barriers in appropriate places to measure length of pipeline stall · 8e922d5b
  Deepak Narayanan authored Jan 09, 2021
  
  8e922d5b
- Interleaved pipeline execution and code refactoring · dd889062
  Deepak Narayanan authored Dec 12, 2020
```
- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions
```
  dd889062
08 Feb, 2021 2 commits
- Merge branch 'ckpt_rng' into 'main' · c1faa9fe
  Mohammad Shoeybi authored Feb 08, 2021
```
Improve handling of rng states in checkpoints.

See merge request ADLR/megatron-lm!231
```
  c1faa9fe
- Improve handling of rng states in checkpoints. · 08a848c7
  Jared Casper authored Feb 08, 2021
  
  08a848c7
06 Feb, 2021 1 commit

Merge branch 'main_fix' into 'main' · 8863af8c

Jared Casper authored Feb 05, 2021

Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API

See merge request ADLR/megatron-lm!230

8863af8c

05 Feb, 2021 3 commits
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 7ffea978
  Deepak Narayanan authored Feb 05, 2021
  
  7ffea978
- Merge branch 'fused_kernel_cond' into 'main' · 2096d356
  Jared Casper authored Feb 05, 2021
```
conditioning fused kernels

See merge request ADLR/megatron-lm!228
```
  2096d356
- address review comments · 0cb36de2
  Vijay Korthikanti authored Feb 05, 2021
  
  0cb36de2
04 Feb, 2021 4 commits
- conditioning fused kernels · 4916bae6
  Vijay Korthikanti authored Feb 04, 2021
  
  4916bae6
- updaded examples/pretrain_ict.sh · f7d96a69
  Mostofa Patwary authored Feb 03, 2021
  
  f7d96a69
- updaded examples/pretrain_ict.sh · 27f0f0ef
  Mostofa Patwary authored Feb 03, 2021
  
  27f0f0ef
- added reload_model_params function · 2cd5f18a
  Mostofa Patwary authored Feb 03, 2021
  
  2cd5f18a