Commits · 0fa7175f0936db7fbe303ca47b25fafca49ef032 · OpenDAS / Megatron-LM

19 Mar, 2021 1 commit
- Bfloat fused softmax + fused layer norm · 0fa7175f
  Mohammad Shoeybi authored Mar 19, 2021
  
  0fa7175f
18 Mar, 2021 2 commits
- Merge branch 'ninja_compilation_fix' into 'main' · d9b1c681
  Jared Casper authored Mar 18, 2021
```
refactored the fused kernels build

See merge request ADLR/megatron-lm!250
```
  d9b1c681
- refactored the fused kernels build · 0d5188c1
  Mohammad Shoeybi authored Mar 17, 2021
  
  0d5188c1
17 Mar, 2021 2 commits
- Merge branch 'softmax_perf' into 'main' · 876096d5
  Jared Casper authored Mar 17, 2021
```
softmax data load/store optimization

See merge request ADLR/megatron-lm!249
```
  876096d5
- fixes to upper triangular masked softmax fusion kernel · 3b12ab15
  Vijay Korthikanti authored Mar 16, 2021
  
  3b12ab15
16 Mar, 2021 2 commits
- minor fixes · 531152d9
  Vijay Korthikanti authored Mar 16, 2021
  
  531152d9
- softmax data load/store optimization · b1a83375
  Vijay Korthikanti authored Mar 16, 2021
  
  b1a83375
08 Mar, 2021 2 commits
- Merge branch 'bfloat_with_fp32_grad_acc' into 'main' · 1a2cb60c
  Jared Casper authored Mar 08, 2021
```
Bfloat with fp32 grad acc

See merge request ADLR/megatron-lm!247
```
  1a2cb60c
- Bfloat with fp32 grad acc · b4bc51b1
  Mohammad Shoeybi authored Mar 08, 2021
  
  b4bc51b1
04 Mar, 2021 6 commits
- Merge branch 'rc-debug-underflow' into 'main' · 87b8b9dc
  Jared Casper authored Mar 04, 2021
```
Rc debug underflow

See merge request ADLR/megatron-lm!246
```
  87b8b9dc
- Fix syntax · 57437cb1
  Rewon Child authored Mar 04, 2021
  
  57437cb1
- Update argument names and fix merge error · 0aff3629
  Rewon Child authored Mar 04, 2021
  
  0aff3629
- Merge main · 41a64613
  Rewon Child authored Mar 04, 2021
  
  41a64613
- Update comments · 8676baca
  Rewon Child authored Mar 04, 2021
  
  8676baca
- Merge branch 'main' into rc-debug-underflow · 14c85e64
  Rewon Child authored Mar 04, 2021
  
  14c85e64
03 Mar, 2021 6 commits
- Merge branch 'main_retriver_merge_ict_eval' into 'main' · 83d26f03
  Jared Casper authored Mar 03, 2021
```
Retrieval index building

See merge request ADLR/megatron-lm!239
```
  83d26f03
- Merge branch 'remove_ring_exchange' into 'main' · 788b59e7
  Jared Casper authored Mar 03, 2021
```
Get PyTorch batched communication API working for interleaved schedule

See merge request ADLR/megatron-lm!242
```
  788b59e7
- Merge branch 'remove_stall_timing' into 'main' · 1acac4e3
  Jared Casper authored Mar 03, 2021
```
Remove pipeline stall timing

See merge request ADLR/megatron-lm!244
```
  1acac4e3
- Remove pipeline stall timing (and avoid Explorer perf regressions) · ab09d819
  Deepak Narayanan authored Mar 03, 2021
  
  ab09d819
- Bring back call to ring_exchange() in _communicate() · cbad126a
  Deepak Narayanan authored Mar 03, 2021
  
  cbad126a
- Get PyTorch batched communication API working for interleaved schedule · 78cf869f
  Deepak Narayanan authored Feb 25, 2021
  
  78cf869f
27 Feb, 2021 1 commit

Merge branch 'deadlock_fix' into 'main' · 3cbf7547

Jared Casper authored Feb 27, 2021

Fix deadlock when get_num_microbatches() < pipeline-parallel size

See merge request ADLR/megatron-lm!243

3cbf7547

26 Feb, 2021 3 commits
- Fix deadlock when get_num_microbatches() < pipeline-parallel size (don't try... · 08ddde0c
  Deepak Narayanan authored Feb 26, 2021
```
Fix deadlock when get_num_microbatches() < pipeline-parallel size (don't try to measure pipeline stall)
```
  08ddde0c
- Merge branch 'torch_ddp' into 'main' · 854c3409
  Jared Casper authored Feb 26, 2021
```
Support Torch DDP for single-stage, num_microbatches() > 1

See merge request ADLR/megatron-lm!240
```
  854c3409
- Support Torch DDP for single-stage, num_microbatches() > 1 · 65eeb427
  Deepak Narayanan authored Feb 24, 2021
  
  65eeb427
25 Feb, 2021 3 commits
- Merge branch 'no-realm-import' into 'main' · 8cd16667
  Mohammad Shoeybi authored Feb 25, 2021
```
Don't import deprecated model from realm_model which is broken.

See merge request ADLR/megatron-lm!241
```
  8cd16667
- Don't import deprecated model from realm_model which is broken. · c6f18ccf
  Jared Casper authored Feb 25, 2021
  
  c6f18ccf
- Merge branch 'kernel_warning' into 'main' · 77d54fbb
  Jared Casper authored Feb 25, 2021
```
fix warning condition

See merge request ADLR/megatron-lm!238
```
  77d54fbb
23 Feb, 2021 6 commits
- Add option to log zeros · 4e77e7c6
  Rewon Child authored Feb 23, 2021
  
  4e77e7c6
- added script for creating embeddings · edf475ff
  Mostofa Patwary authored Feb 23, 2021
  
  edf475ff
- Merge branch 'main' into main_retriver_merge_ict_eval · 60c95ab6
  Mostofa Patwary authored Feb 23, 2021
  
  60c95ab6
- Added code for building embeddings and savings · bcd605f8
  Mostofa Patwary authored Feb 23, 2021
  
  bcd605f8
- fix warning condition · e055f3cf
  Vijay Korthikanti authored Feb 22, 2021
  
  e055f3cf
- Merge branch 'main_dedup' into 'main' · a1f4032b
  Jared Casper authored Feb 22, 2021
```
Storing and loading fingerprints of in deduplication

See merge request ADLR/megatron-lm!236
```
  a1f4032b
22 Feb, 2021 3 commits
- addressed the comments · ebc4a408
  Mostofa Patwary authored Feb 22, 2021
  
  ebc4a408
- Merge branch 'interleaved_schedule_assertion_fix' into 'main' · 90623142
  Jared Casper authored Feb 22, 2021
```
Fix interleaved schedule assertion

See merge request ADLR/megatron-lm!237
```
  90623142
- Fix interleaved schedule assertion · b7067cbd
  Deepak Narayanan authored Feb 22, 2021
  
  b7067cbd
19 Feb, 2021 3 commits
- Merge branch 'main_retriver_merge' into main_retriver_merge_ict_eval · 31d39ec0
  Mostofa Patwary authored Feb 19, 2021
  
  31d39ec0
- updated the readme · 5a867963
  Mostofa Patwary authored Feb 19, 2021
  
  5a867963
- now multiple pickle files can be added · 4b821fb5
  Mostofa Patwary authored Feb 19, 2021
  
  4b821fb5