Commits · 1b3dfa2ff9fe1643e15ddd1cf775abcdb2146f13 · OpenDAS / Megatron-LM

06 Feb, 2021 1 commit
- Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API · 1b3dfa2f
  Deepak Narayanan authored Feb 05, 2021
  
  1b3dfa2f
12 Jan, 2021 2 commits
- Readme update + change gpt2 to gpt · 152aab30
  Mohammad Shoeybi authored Jan 11, 2021
  
  152aab30
- fixed validation loss reporting in tensorboard · 5b74f764
  mohammad authored Jan 11, 2021
  
  5b74f764
07 Jan, 2021 1 commit
- Change some arguments to default to on. · 9acc8956
  Jared Casper authored Jan 06, 2021
  
  9acc8956
06 Jan, 2021 2 commits
- fixed assert issue in clip grads for none grads · 82945c1a
  mohammad authored Jan 05, 2021
  
  82945c1a
- added comment about norm power · 571f10a0
  mohammad authored Jan 05, 2021
  
  571f10a0
05 Jan, 2021 7 commits
- addressed Jareds and Deepaks comments · db88a27b
  mohammad authored Jan 05, 2021
  
  db88a27b
- addressed jareds comments · 7417078a
  mohammad authored Jan 05, 2021
  
  7417078a
- Only create task heads on last pipeline stage. · f772fbc9
  Jared Casper authored Jan 05, 2021
  
  f772fbc9
- some small fixes · 9eedf896
  mohammad authored Jan 04, 2021
  
  9eedf896
- added backward compatibility · 203235fd
  mohammad authored Jan 04, 2021
  
  203235fd
- further refactor, tested, and changed master to main · 22fa9bac
  mohammad authored Jan 04, 2021
  
  22fa9bac
- added reload model params for finetuning · 160ba680
  mohammad authored Jan 04, 2021
  
  160ba680
04 Jan, 2021 1 commit
- Use batched send and recv instead of torch.distributed.ring_exchange() · d899988e
  Deepak Narayanan authored Jan 04, 2021
  
  d899988e
31 Dec, 2020 1 commit
- addressed deepaks comments · 43529f78
  mohammad authored Dec 30, 2020
  
  43529f78
30 Dec, 2020 7 commits
- moved compile helper to initialize · 242770dd
  mshoeybi authored Dec 29, 2020
  
  242770dd
- added clip grads · a495871b
  mohammad authored Dec 29, 2020
  
  a495871b
- done with refactoring and checks · b77d9062
  mohammad authored Dec 29, 2020
  
  b77d9062
- add multi-tensor-apply to clip grad · 9a010310
  mohammad authored Dec 29, 2020
  
  9a010310
- moved grad scaling to multi-tensor-apply · 345f5d0d
  mohammad authored Dec 29, 2020
  
  345f5d0d
- further refactor, matching old results · e68ea218
  mohammad authored Dec 29, 2020
  
  e68ea218
- slight refactor of clip grads · b0a3fdfe
  mohammad authored Dec 29, 2020
  
  b0a3fdfe
28 Dec, 2020 1 commit
- found a bug in l2 norm calculation · 6191ff59
  Mohammad authored Dec 27, 2020
  
  6191ff59
27 Dec, 2020 2 commits
- moved module to model and removed fp16 · b84d7a90
  mohammad authored Dec 26, 2020
  
  b84d7a90
- load and save state dicts added · 97ba5c0e
  mohammad authored Dec 26, 2020
  
  97ba5c0e
25 Dec, 2020 7 commits
- further refactoring · 0888a3e1
  mohammad authored Dec 25, 2020
  
  0888a3e1
- grads is removed from mpu · dfd8ed47
  mohammad authored Dec 25, 2020
  
  dfd8ed47
- clip grad fixed and moved to optimizer · 46879674
  mohammad authored Dec 25, 2020
  
  46879674
- moved entire optimizer build and tested · 28062e14
  mohammad authored Dec 24, 2020
  
  28062e14
- megatron optimizer tested, before working on clip grad · fb218c9d
  mohammad authored Dec 24, 2020
  
  fb218c9d
- fp32 is also working · 2eaa3ccc
  mohammad authored Dec 24, 2020
  
  2eaa3ccc
- working on the new optimizer · c6a58e41
  mohammad authored Dec 24, 2020
  
  c6a58e41
23 Dec, 2020 1 commit
- Checkpoint should be saved only after evaluation pass is run to make sure... · 13bde16f
  Deepak Narayanan authored Dec 23, 2020
```
Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint
```
  13bde16f
22 Dec, 2020 3 commits
- Address Deepak's comments · 83671bbf
  mshoeybi authored Dec 21, 2020
  
  83671bbf
- Add residual connection in fp32 machinery · 8bed1d63
  mohammad authored Dec 21, 2020
  
  8bed1d63
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
19 Dec, 2020 4 commits
- Added a comment to justify 80 percent · 25c07e14
  mohammad authored Dec 14, 2020
  
  25c07e14
- Move args writer to the beginning of training · 6e9d5cb0
  mohammad authored Dec 12, 2020
  
  6e9d5cb0
- Cleaned up load/save checkpoint printing · 8a6e56b8
  mohammad authored Dec 12, 2020
  
  8a6e56b8
- Fix TensorBoard writes · b81cad66
  mohammad authored Dec 12, 2020
  
  b81cad66