Commits · 371d2ea9dad262632ebc129c07ca91d738bb4948 · OpenDAS / Megatron-LM

26 Mar, 2020 4 commits
- Complete definition of ICTBertModel · 371d2ea9
  Neel Kant authored Mar 26, 2020
  
  371d2ea9
- Reformat parts of BertModel · 9873a8da
  Neel Kant authored Mar 26, 2020
  
  9873a8da
- Create ICTBertModel and update model/.__init__.py · fd33e930
  Neel Kant authored Mar 26, 2020
  
  fd33e930
- Add ICT-related parameters to BertModel · bcb320ee
  Neel Kant authored Mar 26, 2020
  
  bcb320ee
24 Mar, 2020 4 commits
- Correct some args and create pretrain_bert_ict.py · 21a916b1
  Neel Kant authored Mar 24, 2020
  
  21a916b1
- Complete __getitem__ for InverseClozeDataset · d2eabecb
  Neel Kant authored Mar 24, 2020
  
  d2eabecb
- Create class InverseClozeTask from bert_sentencepair_dataset and write get_input_and_context · 1c4e8955
  Neel Kant authored Mar 23, 2020
  
  1c4e8955
- Remove unused code · 57f4a8a9
  Neel Kant authored Mar 23, 2020
  
  57f4a8a9
11 Mar, 2020 2 commits
- Merge branch 'memory_optimization' into 'master' · 70174ae3
  Raul Puri authored Mar 10, 2020
```
memory optimization in mpu cross entropy

See merge request ADLR/megatron-lm!32
```
  70174ae3
- memory optimization in mpu cross entropy · 57064fd6
  Mohammad Shoeybi authored Mar 10, 2020
  
  57064fd6
10 Feb, 2020 2 commits
- Merge branch 'mp_merger' into 'master' · ca8dd4ac
  Jared Casper authored Feb 10, 2020
```
Model parallel merger

See merge request ADLR/megatron-lm!28
```
  ca8dd4ac
- Model parallel merger · 57c2060f
  Mohammad Shoeybi authored Feb 10, 2020
  
  57c2060f
04 Feb, 2020 3 commits
- Merge branch 'fixfp16gensamples' into 'master' · 5df85022
  Mohammad Shoeybi authored Feb 04, 2020
```
fixed a bug on fp16 while generating samples

See merge request ADLR/megatron-lm!29
```
  5df85022
- fixed a bug on fp16 while generating samples · cfc6924b
  Mostofa Patwary authored Feb 04, 2020
  
  cfc6924b
- Merge branch 'scale_qkt_exp_lr' into 'master' · a4cb4153
  Jared Casper authored Feb 04, 2020
```
Scale Q*K (query times key) by 1/layer-number and add exponential decay option

See merge request ADLR/megatron-lm!27
```
  a4cb4153
17 Jan, 2020 2 commits
- Merge branch 'model_parallel_generation' into 'master' · 4ad5717e
  Raul Puri authored Jan 16, 2020
```
force output gathering

See merge request ADLR/megatron-lm!17
```
  4ad5717e
- undid padding merge · 7a6d630e
  Raul Puri authored Jan 16, 2020
  
  7a6d630e
14 Jan, 2020 3 commits
- added query-key layer scaling and softmax fp32 option · 6c2d0337
  Mohammad Shoeybi authored Jan 13, 2020
  
  6c2d0337
- added query-key layer scaling and softmax fp32 option · 691747b1
  Mohammad Shoeybi authored Jan 13, 2020
  
  691747b1
- exponential learning rate decay added · 7a3b4c15
  Mohammad Shoeybi authored Jan 13, 2020
  
  7a3b4c15
06 Jan, 2020 1 commit
- Merge branch 'mo_albert_data_loader' into 'master' · 2d76d065
  Jared Casper authored Jan 06, 2020
```
New data loader

See merge request ADLR/megatron-lm!16
```
  2d76d065
27 Dec, 2019 1 commit
- checked and bert, gpt, and albert albert run · f86bb671
  Mohammad Shoeybi authored Dec 26, 2019
  
  f86bb671
26 Dec, 2019 3 commits
- fixed gpt-2 dataloder · d6485684
  Mohammad Shoeybi authored Dec 26, 2019
  
  d6485684
- reverted init in data_utils · b8e0129f
  Mohammad Shoeybi authored Dec 26, 2019
  
  b8e0129f
- changed long seq length to 512 · 43dd21f7
  Mohammad Shoeybi authored Dec 26, 2019
  
  43dd21f7
25 Dec, 2019 1 commit
- added filtering based on sentence length · 2c58c9b0
  Mohammad Shoeybi authored Dec 24, 2019
  
  2c58c9b0
22 Dec, 2019 1 commit
- force output gathering · b9b6fe0d
  Raul Puri authored Dec 22, 2019
  
  b9b6fe0d
04 Dec, 2019 1 commit
- minor cleaning · 75a86a1d
  Mohammad Shoeybi authored Dec 03, 2019
  
  75a86a1d
02 Dec, 2019 1 commit
- Merge branch 'master' into mo_albert_data_loader · 5fc301aa
  Mohammad Shoeybi authored Dec 02, 2019
  
  5fc301aa
27 Nov, 2019 2 commits
- Merge branch 'refactor_gpt2' into 'master' · 1b290993
  Mostofa Patwary authored Nov 27, 2019
```
Refactor gpt2

See merge request ADLR/megatron-lm!15
```
  1b290993
- refactoring · f223ff67
  Mohammad Shoeybi authored Nov 27, 2019
  
  f223ff67
24 Nov, 2019 4 commits
- removed split dataset · 8179ebd3
  Mohammad Shoeybi authored Nov 24, 2019
  
  8179ebd3
- minor fix · 5300c69f
  Mohammad Shoeybi authored Nov 24, 2019
  
  5300c69f
- resolved reproducibity issue · e2add0fd
  Mohammad Shoeybi authored Nov 24, 2019
  
  e2add0fd
- zero worker seems to be working · 0601702a
  Mohammad Shoeybi authored Nov 24, 2019
  
  0601702a
21 Nov, 2019 1 commit
- fixed padding issue · f6a6811f
  Mohammad Shoeybi authored Nov 20, 2019
  
  f6a6811f
20 Nov, 2019 2 commits
- added write option for index map · 9ea9d50f
  Mohammad Shoeybi authored Nov 20, 2019
  
  9ea9d50f
- fixed couple of bugs in helpers · 690291dd
  Mohammad Shoeybi authored Nov 20, 2019
  
  690291dd
19 Nov, 2019 1 commit
- Plumbing for new albert dataset, including mods to arguments for data loaders. · 1a1cfcff
  Jared Casper authored Nov 18, 2019
  
  1a1cfcff
13 Nov, 2019 1 commit
- Plugging new dataset stuff together. · 0058b1a5
  Jared Casper authored Nov 13, 2019
```
Some cleanup of types, etc. in the helpers cpp code.
```
  0058b1a5