Commits · f5eac3d19b89991d2e2c88f50efeb0fe5cc69a45 · OpenDAS / Megatron-LM

19 Dec, 2020 1 commit
- Add pipelining to GLUE and RACE tasks · caa9dca5
  Jared Casper authored Nov 30, 2020
  
  caa9dca5
13 Nov, 2020 1 commit
- New example scripts showing how to use MP, and some notes to main README · a5a41922
  Deepak Narayanan authored Nov 13, 2020
  
  a5a41922
12 Nov, 2020 3 commits
- Address Jared's comments in README and loss_scaler.py · dd079406
  Deepak Narayanan authored Oct 27, 2020
  
  dd079406
- Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
  Deepak Narayanan authored Oct 20, 2020
  
  52a5f2f2
- Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90
  Deepak Narayanan authored Aug 29, 2020
```
Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments
```
  7abd3e90
11 Sep, 2020 1 commit
- Update scaling numbers in README and other small tweaks. · 52819194
  Jared Casper authored Sep 11, 2020
  
  52819194
23 Jul, 2020 1 commit
- Add more documentation · c2a32e12
  Neel Kant authored Jul 22, 2020
  
  c2a32e12
07 Jul, 2020 1 commit
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
29 Jun, 2020 1 commit
- Update README · 68283d93
  Neel Kant authored Jun 28, 2020
  
  68283d93
24 Jun, 2020 2 commits
- Update README · 7b3baaaa
  Neel Kant authored Jun 24, 2020
  
  7b3baaaa
- Update README · 47996737
  Neel Kant authored Jun 24, 2020
  
  47996737
15 May, 2020 1 commit
- update · faa462fa
  Evelina Bakhturina authored May 14, 2020
```
Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>
```
  faa462fa
14 May, 2020 4 commits
- update · 5f871088
  Evelina Bakhturina authored May 14, 2020
```
Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>
```
  5f871088
- update · 1f39e69b
  Evelina Bakhturina authored May 14, 2020
```
Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>
```
  1f39e69b
- update2 · 73fef4fc
  Evelina Bakhturina authored May 14, 2020
```
Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>
```
  73fef4fc
- six, regex, apex added to requirements · c15829b5
  Evelina Bakhturina authored May 14, 2020
```
Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>
```
  c15829b5
05 May, 2020 1 commit
- Fix typos · 4e2b7b26
  David E. Weekly authored May 05, 2020
```
Fixed some small typos in the README
```
  4e2b7b26
21 Apr, 2020 1 commit
- Update README.md · 89462697
  Raul Puri authored Apr 20, 2020
  
  89462697
20 Apr, 2020 3 commits
- Move links to download checkpoints section · b46e28bd
  Steven Steinke authored Apr 20, 2020
  
  b46e28bd
- Links to vocab in Data Preprocessing; added "WordPiece" to description · 8e4cb2a6
  Steven Steinke authored Apr 20, 2020
  
  8e4cb2a6
- Links to vocabs and merge table in Data Preprocessing · bdc3d5ba
  Steven Steinke authored Apr 20, 2020
  
  bdc3d5ba
16 Apr, 2020 9 commits
- remove wikitext comment · 7ec42d95
  Steven Steinke authored Apr 16, 2020
  
  7ec42d95
- RACE explained · 4f770782
  Steven Steinke authored Apr 16, 2020
  
  4f770782
- Update README.md · d976f011
  Steven Steinke authored Apr 16, 2020
  
  d976f011
- Fixed minor inconsistencies in scripts, added distributed comment · 2e0b3fca
  Steven Steinke authored Apr 16, 2020
  
  2e0b3fca
- looks great · 3df23c72
  Mohammad authored Apr 16, 2020
  
  3df23c72
- latest · 0d8f1243
  Mohammad authored Apr 16, 2020
  
  0d8f1243
- updated scripts · ca19a8b5
  Mohammad authored Apr 16, 2020
  
  ca19a8b5
- made monir changes to the first two paras · fe40745e
  Mohammad authored Apr 16, 2020
  
  fe40745e
- moved steves branch · 78cb1781
  Mohammad authored Apr 16, 2020
  
  78cb1781
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e

23 Apr, 2019 1 commit

Faster dataloader merge () · 66719e97

Raul Puri authored Apr 23, 2019

* threaded tf_dl+presplit sentences+shuffled dataset with resume

* elaborate in readme

66719e97

27 Mar, 2019 1 commit
- initial commit · fb4cbdc2
  Raul Puri authored Mar 27, 2019
  
  fb4cbdc2