Commits · 52a5f2f272e5ef242eb271227da712f7dfc55da3 · OpenDAS / Megatron-LM

12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

14 Oct, 2020 2 commits
- added adam betas and eps as arguments · c55e154f
  Mostofa Patwary authored Oct 13, 2020
  
  c55e154f
- added adam betas and eps as arguments · 48269d8d
  Mostofa Patwary authored Oct 13, 2020
  
  48269d8d
30 Sep, 2020 1 commit
- checkpoint versioning · a8d47812
  Vijay Korthikanti authored Sep 29, 2020
  
  a8d47812
29 Sep, 2020 1 commit
- support for old checkpoint format · 37ae6646
  Vijay Korthikanti authored Sep 28, 2020
  
  37ae6646
18 Sep, 2020 1 commit
- scaling-generalized masking-softmax fusion kernel · d73def35
  root authored Sep 18, 2020
  
  d73def35
11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
02 Sep, 2020 2 commits
- addressed Jareds comments · 5d29769c
  mohammad authored Sep 02, 2020
  
  5d29769c
- added splitting checkpointed activations · d6c4248b
  mohammad authored Sep 02, 2020
  
  d6c4248b
28 Aug, 2020 1 commit
- Added documentation · b6b48a32
  Boris Fomitchev authored Aug 27, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  b6b48a32
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
25 Aug, 2020 1 commit
- Making onnx-safe option overrideable · d4bb6cb0
  Boris Fomitchev authored Aug 25, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  d4bb6cb0
06 Aug, 2020 1 commit
- Implementing lazy parallel initialization · 5c04ceb3
  Boris Fomitchev authored Aug 05, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  5c04ceb3
03 Aug, 2020 1 commit
- added gpu initialization and option to avoid master values · 0403b808
  mohammad authored Aug 03, 2020
  
  0403b808
28 Jul, 2020 1 commit
- Replacing --erf-gelu option with explicit --onnx-safe option · fe794c5a
  Boris Fomitchev authored Jul 27, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  fe794c5a
24 Jul, 2020 1 commit
- ONNX export fix, including one for gelu() · 158a99c3
  Boris Fomitchev authored Jul 23, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  158a99c3
23 Jul, 2020 1 commit
- Address comments from last week · 11f76cd3
  Neel Kant authored Jul 22, 2020
  
  11f76cd3
07 Jul, 2020 2 commits
- Made topk accuracy reporting optional · de6640be
  Neel Kant authored Jul 07, 2020
  
  de6640be
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
02 Jul, 2020 1 commit
- One sentence options · 03feecbc
  Neel Kant authored Jul 02, 2020
  
  03feecbc
29 Jun, 2020 1 commit
- Changes after running on draco · 44860f8d
  Neel Kant authored Jun 28, 2020
  
  44860f8d
25 Jun, 2020 1 commit
- Add stanza NER salient span masking · e7045139
  Neel Kant authored Jun 24, 2020
  
  e7045139
22 Jun, 2020 1 commit
- Address most of comments from code review w/ Mohammad · 6495927e
  Neel Kant authored Jun 22, 2020
  
  6495927e
15 Jun, 2020 1 commit
- Debugging done on Circe · b8bb0b49
  Mohammad authored Jun 15, 2020
  
  b8bb0b49
09 Jun, 2020 1 commit
- Correct retrieval utility and add salient span preprocessing · bf599e86
  Neel Kant authored Jun 08, 2020
  
  bf599e86
05 Jun, 2020 3 commits
- addressed jareds comments · 197c132e
  mohammad authored Jun 05, 2020
  
  197c132e
- added fp16 cross entropy loss option for gpt2 · acfe848e
  mohammad authored Jun 05, 2020
  
  acfe848e
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
27 May, 2020 1 commit
- Correct CrossEntropyLoss · 0e8f4331
  Neel Kant authored May 27, 2020
  
  0e8f4331
24 May, 2020 1 commit
- Use Faiss GPU index and report retrieval utility · 8573ab35
  Neel Kant authored May 24, 2020
  
  8573ab35
19 May, 2020 1 commit
- Indexer_async works in theory · e338e311
  Neel Kant authored May 18, 2020
  
  e338e311
14 May, 2020 1 commit
- Add retrieval utility and autoresume for indexer · 2f7d666c
  Neel Kant authored May 14, 2020
  
  2f7d666c
07 May, 2020 1 commit
- Resolve internal merge conflict · 0f5e2809
  Neel Kant authored May 06, 2020
  
  0f5e2809
06 May, 2020 2 commits
- tested and seems to be working · b1ac9fd3
  mohammad authored May 05, 2020
  
  b1ac9fd3
- added parameters sharing · 80f90dcd
  Mohammad authored May 05, 2020
  
  80f90dcd
05 May, 2020 1 commit
- Refactor and add more REALM arguments · 730266ca
  Neel Kant authored May 05, 2020
  
  730266ca
03 May, 2020 1 commit
- Correct arguments · 002cb170
  Neel Kant authored May 03, 2020
  
  002cb170
30 Apr, 2020 1 commit
- Whitening code · 9d225b44
  Neel Kant authored Apr 29, 2020
  
  9d225b44
27 Apr, 2020 1 commit
- remoed abbreviation matching from argparser · cf595489
  Mohammad authored Apr 27, 2020
  
  cf595489