Commits · b84d7a90122a2fb93793492b3ebefdd39c69a12e · OpenDAS / Megatron-LM

27 Dec, 2020 1 commit
- moved module to model and removed fp16 · b84d7a90
  mohammad authored Dec 26, 2020
  
  b84d7a90
12 Nov, 2020 5 commits
- Refactor word_embeddings_weight() logic into separate method, and other Mohammad comments · 57c3b364
  Deepak Narayanan authored Nov 03, 2020
  
  57c3b364
- Clarifications in comments and minor refactoring to make main training loop more readable · 8fb2bc8c
  Deepak Narayanan authored Oct 28, 2020
  
  8fb2bc8c
- Removal of unneeded changes so that diff is smaller · 275d4e64
  Deepak Narayanan authored Oct 20, 2020
  
  275d4e64
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
05 Jun, 2020 4 commits
- addressed jareds comments · 197c132e
  mohammad authored Jun 05, 2020
  
  197c132e
- added fp16 cross entropy loss option for gpt2 · 22e3c7e6
  mohammad authored Jun 05, 2020
  
  22e3c7e6
- added fp16 cross entropy loss option for gpt2 · acfe848e
  mohammad authored Jun 05, 2020
  
  acfe848e
- testing · 2ede8235
  mohammad authored Jun 05, 2020
  
  2ede8235
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
15 Apr, 2020 1 commit
- changed gpt2 masking to binary and masked_fill · 8ceed7c7
  Mohammad authored Apr 14, 2020
  
  8ceed7c7
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
03 Apr, 2020 1 commit
- working on refactoring text generation · ce29d4d5
  Mohammad authored Apr 02, 2020
  
  ce29d4d5
31 Mar, 2020 1 commit
- arguments in the model refactored · deffcb6a
  Mohammad authored Mar 31, 2020
  
  deffcb6a
14 Jan, 2020 1 commit
- added query-key layer scaling and softmax fp32 option · 691747b1
  Mohammad Shoeybi authored Jan 13, 2020
  
  691747b1
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290