Commits · b69bc7efd69d3f8fde742bb57feb3c13bd66c5fd · OpenDAS / Megatron-LM

04 Jan, 2021 1 commit
- Use batched send and recv instead of torch.distributed.ring_exchange() · d899988e
  Deepak Narayanan authored Jan 04, 2021
  
  d899988e
22 Dec, 2020 3 commits
- Address Deepak's comments · 83671bbf
  mshoeybi authored Dec 21, 2020
  
  83671bbf
- Add residual connection in fp32 machinery · 8bed1d63
  mohammad authored Dec 21, 2020
  
  8bed1d63
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
19 Dec, 2020 10 commits
- Fix some bugs, add exit-duration capability · a31833ce
  mshoeybi authored Dec 11, 2020
  
  a31833ce
- Nicer error messages for deprecated arguments · 2623551d
  Jared Casper authored Dec 10, 2020
  
  2623551d
- Change lr-warmup-percent to lr-warmup-fraction · 9321d5c6
  Jared Casper authored Dec 10, 2020
  
  9321d5c6
- Work batch-size name changes into task code · 3afcba6e
  Jared Casper authored Dec 09, 2020
  
  3afcba6e
- Address Jared's comments · 00ac56ab
  mohammad authored Dec 09, 2020
  
  00ac56ab
- Sample based learning rate computation · 22ab91bb
  mohammad authored Dec 08, 2020
  
  22ab91bb
- Minor refactoring · c30ba0f7
  mohammad authored Dec 08, 2020
  
  c30ba0f7
- Add constant num micro-batches calculator · feecd5d9
  mohammad authored Dec 07, 2020
  
  feecd5d9
- Add micro-batch size calculator · 6ea23928
  mohammad authored Dec 06, 2020
  
  6ea23928
- Rename --batch-size to --micro-batch-size and drop in-minibatch from... · 9019bbf4
  mohammad authored Dec 06, 2020
```
Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch
```
  9019bbf4
02 Dec, 2020 2 commits
- addressed Jareds comments · fa80af26
  mohammad authored Dec 02, 2020
  
  fa80af26
- addressed Jareds comments · 98989693
  mohammad authored Dec 02, 2020
  
  98989693
29 Nov, 2020 1 commit
- implemented blending datasets · 65290033
  mohammad authored Nov 28, 2020
  
  65290033
28 Nov, 2020 1 commit
- added consumed tokens to checkpoints and some refactoring · f0a445fa
  mohammad authored Nov 27, 2020
  
  f0a445fa
12 Nov, 2020 5 commits
- Throw exception if ring_exchange is not available when pipeline_model_parallel_size > 1 · 2d8de296
  Deepak Narayanan authored Oct 30, 2020
  
  2d8de296
- Remove unused parameter sharing logic · 1271fd73
  Deepak Narayanan authored Oct 28, 2020
  
  1271fd73
- Refactor communication code in main training loop to helper method · 318d68c2
  Deepak Narayanan authored Oct 23, 2020
  
  318d68c2
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
20 Oct, 2020 1 commit
- Change distributed argument group title to appropriate one. · c147a3e8
  Kazuhiro Yamasaki authored Oct 20, 2020
  
  c147a3e8
14 Oct, 2020 2 commits
- added adam betas and eps as arguments · c55e154f
  Mostofa Patwary authored Oct 13, 2020
  
  c55e154f
- added adam betas and eps as arguments · 48269d8d
  Mostofa Patwary authored Oct 13, 2020
  
  48269d8d
08 Oct, 2020 1 commit
- Escape % in --split help message · e629f36e
  Heungsub Lee authored Oct 09, 2020
  
  e629f36e
30 Sep, 2020 1 commit
- checkpoint versioning · a8d47812
  Vijay Korthikanti authored Sep 29, 2020
  
  a8d47812
29 Sep, 2020 1 commit
- support for old checkpoint format · 37ae6646
  Vijay Korthikanti authored Sep 28, 2020
  
  37ae6646
18 Sep, 2020 1 commit
- scaling-generalized masking-softmax fusion kernel · d73def35
  root authored Sep 18, 2020
  
  d73def35
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
28 Aug, 2020 1 commit
- Added documentation · b6b48a32
  Boris Fomitchev authored Aug 27, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  b6b48a32
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
25 Aug, 2020 1 commit
- Making onnx-safe option overrideable · d4bb6cb0
  Boris Fomitchev authored Aug 25, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  d4bb6cb0
06 Aug, 2020 1 commit
- Implementing lazy parallel initialization · 5c04ceb3
  Boris Fomitchev authored Aug 05, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  5c04ceb3
03 Aug, 2020 1 commit
- added gpu initialization and option to avoid master values · 0403b808
  mohammad authored Aug 03, 2020
  
  0403b808
28 Jul, 2020 1 commit
- Replacing --erf-gelu option with explicit --onnx-safe option · fe794c5a
  Boris Fomitchev authored Jul 27, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  fe794c5a
24 Jul, 2020 1 commit
- ONNX export fix, including one for gelu() · 158a99c3
  Boris Fomitchev authored Jul 23, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  158a99c3