Commits · f5eac3d19b89991d2e2c88f50efeb0fe5cc69a45 · OpenDAS / Megatron-LM

27 Dec, 2020 1 commit
- moved module to model and removed fp16 · b84d7a90
  mohammad authored Dec 26, 2020
  
  b84d7a90
22 Dec, 2020 2 commits
- Address Deepak's comments · 83671bbf
  mshoeybi authored Dec 21, 2020
  
  83671bbf
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
12 Nov, 2020 6 commits
- Refactor code according to Jared's comments: move pipelining and... · 1979c242
  Deepak Narayanan authored Nov 12, 2020
```
Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size
```
  1979c242
- Only transpose hidden_states when necessary · 6abf39be
  Deepak Narayanan authored Nov 03, 2020
  
  6abf39be
- Refactor word_embeddings_weight() logic into separate method, and other Mohammad comments · 57c3b364
  Deepak Narayanan authored Nov 03, 2020
  
  57c3b364
- Remove unused parameter sharing logic · 1271fd73
  Deepak Narayanan authored Oct 28, 2020
  
  1271fd73
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
13 Oct, 2020 2 commits
- typo fix · 01dffdb5
  Vijay Korthikanti authored Oct 12, 2020
  
  01dffdb5
- address review comments · 0bc75448
  Vijay Korthikanti authored Oct 12, 2020
  
  0bc75448
12 Oct, 2020 2 commits
- Incrementing checkpoint version to 2.0 · 9b0083ea
  Vijay Korthikanti authored Oct 12, 2020
  
  9b0083ea
- reordering perf fix · ee327acd
  Vijay Korthikanti authored Oct 12, 2020
  
  ee327acd
09 Oct, 2020 1 commit
- support for different query key sequence lengths · d763f54a
  Vijay Korthikanti authored Oct 08, 2020
  
  d763f54a
30 Sep, 2020 1 commit
- checkpoint versioning · a8d47812
  Vijay Korthikanti authored Sep 29, 2020
  
  a8d47812
29 Sep, 2020 2 commits
- bug fix · 7d4ad51e
  Vijay Korthikanti authored Sep 29, 2020
  
  7d4ad51e
- support for old checkpoint format · 37ae6646
  Vijay Korthikanti authored Sep 28, 2020
  
  37ae6646
18 Sep, 2020 1 commit
- scaling-generalized masking-softmax fusion kernel · d73def35
  root authored Sep 18, 2020
  
  d73def35
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
02 Sep, 2020 1 commit
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
26 Jul, 2020 1 commit
- Complete fix for APEX absence in NeMo · aa0ee72e
  Boris Fomitchev authored Jul 25, 2020
  
  aa0ee72e
24 Jul, 2020 2 commits
- Fixed APEX test · 8f0af7cd
  Boris Fomitchev authored Jul 24, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  8f0af7cd
- Warning added · b04eb0a2
  Boris Fomitchev authored Jul 23, 2020
  
  b04eb0a2
22 Jul, 2020 1 commit
- Initialization fixes: allowing simple case like pytest pass, also making apex optional · 9026b86d
  Boris Fomitchev authored Jul 22, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  9026b86d
06 May, 2020 4 commits
- addressed jareds comment · c9c69c1e
  mohammad authored May 06, 2020
  
  c9c69c1e
- tested and seems to be working · b1ac9fd3
  mohammad authored May 05, 2020
  
  b1ac9fd3
- added parameters sharing · f257d0ee
  Mohammad authored May 05, 2020
  
  f257d0ee
- added parameters sharing · 80f90dcd
  Mohammad authored May 05, 2020
  
  80f90dcd
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
10 Apr, 2020 1 commit
- fp32 fixes · 5f174c07
  Mohammad authored Apr 10, 2020
  
  5f174c07
31 Mar, 2020 1 commit
- arguments in the model refactored · deffcb6a
  Mohammad authored Mar 31, 2020
  
  deffcb6a
10 Feb, 2020 1 commit
- Model parallel merger · 57c2060f
  Mohammad Shoeybi authored Feb 10, 2020
  
  57c2060f
14 Jan, 2020 2 commits
- added query-key layer scaling and softmax fp32 option · 6c2d0337
  Mohammad Shoeybi authored Jan 13, 2020
  
  6c2d0337
- added query-key layer scaling and softmax fp32 option · 691747b1
  Mohammad Shoeybi authored Jan 13, 2020
  
  691747b1
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290