Commits · 52a5f2f272e5ef242eb271227da712f7dfc55da3 · OpenDAS / Megatron-LM

12 Nov, 2020 3 commits
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
- fp32 working · 28cd66e1
  mohammad authored Aug 28, 2020
  
  28cd66e1
03 Nov, 2020 1 commit
- fixed loss average when all but one value is skipped · 664cd28b
  mohammad authored Nov 03, 2020
  
  664cd28b
14 Oct, 2020 2 commits
- added adam betas and eps as arguments · c55e154f
  Mostofa Patwary authored Oct 13, 2020
  
  c55e154f
- added adam betas and eps as arguments · 48269d8d
  Mostofa Patwary authored Oct 13, 2020
  
  48269d8d
13 Oct, 2020 3 commits
- suppress kernel loading output · 8fca9b49
  Vijay Korthikanti authored Oct 12, 2020
  
  8fca9b49
- typo fix · 01dffdb5
  Vijay Korthikanti authored Oct 12, 2020
  
  01dffdb5
- address review comments · 0bc75448
  Vijay Korthikanti authored Oct 12, 2020
  
  0bc75448
12 Oct, 2020 2 commits
- Incrementing checkpoint version to 2.0 · 9b0083ea
  Vijay Korthikanti authored Oct 12, 2020
  
  9b0083ea
- reordering perf fix · ee327acd
  Vijay Korthikanti authored Oct 12, 2020
  
  ee327acd
09 Oct, 2020 1 commit
- support for different query key sequence lengths · d763f54a
  Vijay Korthikanti authored Oct 08, 2020
  
  d763f54a
08 Oct, 2020 1 commit
- Create build directory for fused_kernels before building. · 54ded172
  Jared Casper authored Oct 07, 2020
  
  54ded172
06 Oct, 2020 1 commit
- build directory · 3dcd7dc0
  Vijay Korthikanti authored Oct 05, 2020
  
  3dcd7dc0
05 Oct, 2020 2 commits
- build_directory argument · a7f4e460
  Vijay Korthikanti authored Oct 05, 2020
  
  a7f4e460
- avoid non deterministic arch order in compilation · 37d77ca0
  Vijay Korthikanti authored Oct 05, 2020
  
  37d77ca0
02 Oct, 2020 2 commits
- pip 1.1.5 · f9387ec3
  Evelina authored Oct 02, 2020
```
Signed-off-by: Evelina <ebakhturina@nvidia.com>
```
  f9387ec3
- Fixed uninit communication gruop · 057a1fab
  Mostofa Patwary authored Oct 01, 2020
  
  057a1fab
01 Oct, 2020 1 commit
- fix for nemo: do not initialize mpu if it is already initialized · 6e433055
  mohammad authored Oct 01, 2020
  
  6e433055
30 Sep, 2020 2 commits
- address review comments · 64e45f29
  Vijay Korthikanti authored Sep 29, 2020
  
  64e45f29
- checkpoint versioning · a8d47812
  Vijay Korthikanti authored Sep 29, 2020
  
  a8d47812
29 Sep, 2020 3 commits
- bug fix · 7d4ad51e
  Vijay Korthikanti authored Sep 29, 2020
  
  7d4ad51e
- support for old checkpoint format · 37ae6646
  Vijay Korthikanti authored Sep 28, 2020
  
  37ae6646
- Text generation bugs · 1f3f834c
  Mostofa Patwary authored Sep 28, 2020
  
  1f3f834c
25 Sep, 2020 1 commit
- fixed the interation count · 2db01991
  mohammad authored Sep 24, 2020
  
  2db01991
24 Sep, 2020 1 commit
- nan values report added · abd7636e
  mohammad authored Sep 23, 2020
  
  abd7636e
22 Sep, 2020 1 commit
- fixed logging bug · a74f8d9c
  mohammad authored Sep 22, 2020
  
  a74f8d9c
18 Sep, 2020 1 commit
- scaling-generalized masking-softmax fusion kernel · d73def35
  root authored Sep 18, 2020
  
  d73def35
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
03 Sep, 2020 1 commit
- debuged the cpu init for mp=1 · 04cd82bd
  mohammad authored Sep 03, 2020
  
  04cd82bd
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
01 Sep, 2020 1 commit
- 1.1.4. pip version · e8c47e76
  Evelina authored Sep 01, 2020
```
Signed-off-by: Evelina <ebakhturina@nvidia.com>
```
  e8c47e76
28 Aug, 2020 1 commit
- Added documentation · b6b48a32
  Boris Fomitchev authored Aug 27, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  b6b48a32
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
26 Aug, 2020 1 commit
- Making --lazy_mpu_init act as _USE_CPU_INITIALIZATION · e10760ea
  Boris Fomitchev authored Aug 26, 2020
  
  e10760ea
25 Aug, 2020 1 commit
- Making onnx-safe option overrideable · d4bb6cb0
  Boris Fomitchev authored Aug 25, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  d4bb6cb0
07 Aug, 2020 3 commits
- pip update · 8ad2af6d
  Evelina authored Aug 07, 2020
```
Signed-off-by: Evelina <ebakhturina@nvidia.com>
```
  8ad2af6d
- comment fix · 4e8ace9b
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  4e8ace9b
- fixed typo in comment · 39c80331
  Boris Fomitchev authored Aug 06, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  39c80331