Commits · e68ea2188eee34fab790aa2467846bd68b5d7ee7 · OpenDAS / Megatron-LM

30 Dec, 2020 2 commits
- further refactor, matching old results · e68ea218
  mohammad authored Dec 29, 2020
  
  e68ea218
- slight refactor of clip grads · b0a3fdfe
  mohammad authored Dec 29, 2020
  
  b0a3fdfe
28 Dec, 2020 1 commit
- found a bug in l2 norm calculation · 6191ff59
  Mohammad authored Dec 27, 2020
  
  6191ff59
27 Dec, 2020 2 commits
- moved module to model and removed fp16 · b84d7a90
  mohammad authored Dec 26, 2020
  
  b84d7a90
- load and save state dicts added · 97ba5c0e
  mohammad authored Dec 26, 2020
  
  97ba5c0e
25 Dec, 2020 7 commits
- further refactoring · 0888a3e1
  mohammad authored Dec 25, 2020
  
  0888a3e1
- grads is removed from mpu · dfd8ed47
  mohammad authored Dec 25, 2020
  
  dfd8ed47
- clip grad fixed and moved to optimizer · 46879674
  mohammad authored Dec 25, 2020
  
  46879674
- moved entire optimizer build and tested · 28062e14
  mohammad authored Dec 24, 2020
  
  28062e14
- megatron optimizer tested, before working on clip grad · fb218c9d
  mohammad authored Dec 24, 2020
  
  fb218c9d
- fp32 is also working · 2eaa3ccc
  mohammad authored Dec 24, 2020
  
  2eaa3ccc
- working on the new optimizer · c6a58e41
  mohammad authored Dec 24, 2020
  
  c6a58e41
23 Dec, 2020 1 commit
- Checkpoint should be saved only after evaluation pass is run to make sure... · 13bde16f
  Deepak Narayanan authored Dec 23, 2020
```
Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint
```
  13bde16f
22 Dec, 2020 3 commits
- Address Deepak's comments · 83671bbf
  mshoeybi authored Dec 21, 2020
  
  83671bbf
- Add residual connection in fp32 machinery · 8bed1d63
  mohammad authored Dec 21, 2020
  
  8bed1d63
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
19 Dec, 2020 24 commits
- Added a comment to justify 80 percent · 25c07e14
  mohammad authored Dec 14, 2020
  
  25c07e14
- Move args writer to the beginning of training · 6e9d5cb0
  mohammad authored Dec 12, 2020
  
  6e9d5cb0
- Cleaned up load/save checkpoint printing · 8a6e56b8
  mohammad authored Dec 12, 2020
  
  8a6e56b8
- Fix TensorBoard writes · b81cad66
  mohammad authored Dec 12, 2020
  
  b81cad66
- Fix loss addition in TensorBoard · 5a304ede
  mshoeybi authored Dec 12, 2020
  
  5a304ede
- Some bugfixes · 29a69547
  mshoeybi authored Dec 11, 2020
  
  29a69547
- Last epoch should not be globally shuffled · 39181113
  mshoeybi authored Dec 11, 2020
  
  39181113
- Address Jared's comments · 56243e19
  mshoeybi authored Dec 11, 2020
  
  56243e19
- Fix some bugs, add exit-duration capability · a31833ce
  mshoeybi authored Dec 11, 2020
  
  a31833ce
- Add comment describing _PIPELINE_GLOBAL_RANKS · 51315905
  Jared Casper authored Dec 10, 2020
  
  51315905
- Fix text generation without recompute · 1d4e8760
  Jared Casper authored Dec 10, 2020
  
  1d4e8760
- Nicer error messages for deprecated arguments · 2623551d
  Jared Casper authored Dec 10, 2020
  
  2623551d
- Change lr-warmup-percent to lr-warmup-fraction · 9321d5c6
  Jared Casper authored Dec 10, 2020
  
  9321d5c6
- Work batch-size name changes into task code · 3afcba6e
  Jared Casper authored Dec 09, 2020
  
  3afcba6e
- Initial implementation of pipelined text generation · 5c45db4a
  Jared Casper authored Dec 09, 2020
  
  5c45db4a
- Add pipelining to GLUE and RACE tasks · caa9dca5
  Jared Casper authored Nov 30, 2020
  
  caa9dca5
- Better memory tracking across pipeline-parallel ranks · 3574b8e6
  Deepak Narayanan authored Dec 06, 2020
  
  3574b8e6
- Address Jared's comments · 00ac56ab
  mohammad authored Dec 09, 2020
  
  00ac56ab
- Sample based learning rate computation · 22ab91bb
  mohammad authored Dec 08, 2020
  
  22ab91bb
- Minor fixes for batch size rampup · 6a68502d
  mohammad authored Dec 08, 2020
  
  6a68502d
- Support for ramping up the batch size · de0b70a0
  mohammad authored Dec 08, 2020
  
  de0b70a0
- Minor refactoring · c30ba0f7
  mohammad authored Dec 08, 2020
  
  c30ba0f7
- Add constant num micro-batches calculator · feecd5d9
  mohammad authored Dec 07, 2020
  
  feecd5d9
- Add micro-batch size calculator · 6ea23928
  mohammad authored Dec 06, 2020
  
  6ea23928