Commits · 17d897e001b2d3bc720df8685fd70dc35286c853 · OpenDAS / Megatron-LM

"docs/vscode:/vscode.git/clone" did not exist on "72a98a86c6bc17573cbe61a26061689b3830e5af"

29 Jan, 2021 1 commit
- WIP: main_retriver_merge · 17d897e0
  Mostofa Patwary authored Jan 29, 2021
  
  17d897e0
05 Jan, 2021 1 commit
- Only create task heads on last pipeline stage. · f772fbc9
  Jared Casper authored Jan 05, 2021
  
  f772fbc9
04 Jan, 2021 1 commit
- Use batched send and recv instead of torch.distributed.ring_exchange() · d899988e
  Deepak Narayanan authored Jan 04, 2021
  
  d899988e
23 Dec, 2020 1 commit
- Checkpoint should be saved only after evaluation pass is run to make sure... · 13bde16f
  Deepak Narayanan authored Dec 23, 2020
```
Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint
```
  13bde16f
22 Dec, 2020 3 commits
- Address Deepak's comments · 83671bbf
  mshoeybi authored Dec 21, 2020
  
  83671bbf
- Add residual connection in fp32 machinery · 8bed1d63
  mohammad authored Dec 21, 2020
  
  8bed1d63
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
19 Dec, 2020 26 commits
- Added a comment to justify 80 percent · 25c07e14
  mohammad authored Dec 14, 2020
  
  25c07e14
- Move args writer to the beginning of training · 6e9d5cb0
  mohammad authored Dec 12, 2020
  
  6e9d5cb0
- Cleaned up load/save checkpoint printing · 8a6e56b8
  mohammad authored Dec 12, 2020
  
  8a6e56b8
- Fix TensorBoard writes · b81cad66
  mohammad authored Dec 12, 2020
  
  b81cad66
- Fix loss addition in TensorBoard · 5a304ede
  mshoeybi authored Dec 12, 2020
  
  5a304ede
- Some bugfixes · 29a69547
  mshoeybi authored Dec 11, 2020
  
  29a69547
- Last epoch should not be globally shuffled · 39181113
  mshoeybi authored Dec 11, 2020
  
  39181113
- Address Jared's comments · 56243e19
  mshoeybi authored Dec 11, 2020
  
  56243e19
- Fix some bugs, add exit-duration capability · a31833ce
  mshoeybi authored Dec 11, 2020
  
  a31833ce
- Add comment describing _PIPELINE_GLOBAL_RANKS · 51315905
  Jared Casper authored Dec 10, 2020
  
  51315905
- Fix text generation without recompute · 1d4e8760
  Jared Casper authored Dec 10, 2020
  
  1d4e8760
- Nicer error messages for deprecated arguments · 2623551d
  Jared Casper authored Dec 10, 2020
  
  2623551d
- Change lr-warmup-percent to lr-warmup-fraction · 9321d5c6
  Jared Casper authored Dec 10, 2020
  
  9321d5c6
- Work batch-size name changes into task code · 3afcba6e
  Jared Casper authored Dec 09, 2020
  
  3afcba6e
- Initial implementation of pipelined text generation · 5c45db4a
  Jared Casper authored Dec 09, 2020
  
  5c45db4a
- Add pipelining to GLUE and RACE tasks · caa9dca5
  Jared Casper authored Nov 30, 2020
  
  caa9dca5
- Better memory tracking across pipeline-parallel ranks · 3574b8e6
  Deepak Narayanan authored Dec 06, 2020
  
  3574b8e6
- Address Jared's comments · 00ac56ab
  mohammad authored Dec 09, 2020
  
  00ac56ab
- Sample based learning rate computation · 22ab91bb
  mohammad authored Dec 08, 2020
  
  22ab91bb
- Minor fixes for batch size rampup · 6a68502d
  mohammad authored Dec 08, 2020
  
  6a68502d
- Support for ramping up the batch size · de0b70a0
  mohammad authored Dec 08, 2020
  
  de0b70a0
- Minor refactoring · c30ba0f7
  mohammad authored Dec 08, 2020
  
  c30ba0f7
- Add constant num micro-batches calculator · feecd5d9
  mohammad authored Dec 07, 2020
  
  feecd5d9
- Add micro-batch size calculator · 6ea23928
  mohammad authored Dec 06, 2020
  
  6ea23928
- Rename --batch-size to --micro-batch-size and drop in-minibatch from... · 9019bbf4
  mohammad authored Dec 06, 2020
```
Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch
```
  9019bbf4
- Make an eval iteration the same number of samples as a training iteration · a84a5fa0
  Jared Casper authored Dec 03, 2020
  
  a84a5fa0
03 Dec, 2020 1 commit
- found a bug in consumed tokens initialization · e2a4d426
  mohammad authored Dec 02, 2020
  
  e2a4d426
02 Dec, 2020 3 commits
- addressed Jareds comments · fa80af26
  mohammad authored Dec 02, 2020
  
  fa80af26
- addressed Jareds comments · 98989693
  mohammad authored Dec 02, 2020
  
  98989693
- addrressed jareds comments · cebd3b8b
  mohammad authored Dec 02, 2020
  
  cebd3b8b
30 Nov, 2020 2 commits
- refactored learning rate scheduler so addition of variable batch size is easier · ff12df6b
  mohammad authored Nov 29, 2020
  
  ff12df6b
- added refactored learning rate · 16193619
  mohammad authored Nov 29, 2020
  
  16193619
29 Nov, 2020 1 commit
- implemented blending datasets · 65290033
  mohammad authored Nov 28, 2020
  
  65290033