Commits · c99fa80ce9f99303a07ced0cf3f063eaa34ac407 · wxj / Megatron-LM

"host/online_compile/include/md5.hpp" did not exist on "d2315b0dfcd6f31cca4328819eaf60d77e952dd6"

14 Apr, 2020 9 commits
- Merge branch 'move_openwebtext' into 'staging' · c99fa80c
  Mohammad Shoeybi authored Apr 14, 2020
```
Move openwebtext scripts to tools directory.

See merge request ADLR/megatron-lm!50
```
  c99fa80c
- Move openwebtext scripts to tools directory. · 0dc10d09
  Jared Casper authored Apr 14, 2020
  
  0dc10d09
- Merge branch 'staging_gpt2_dataloader' into 'staging' · 69d8ff1c
  Jared Casper authored Apr 14, 2020
```
GPT2 dataloader using indexed dataset

See merge request ADLR/megatron-lm!45
```
  69d8ff1c
- removed docker directory · 0afe1fde
  Mohammad authored Apr 14, 2020
  
  0afe1fde
- Merge branch 'staging' into staging_gpt2_dataloader · 6ae4a03c
  Mohammad authored Apr 14, 2020
  
  6ae4a03c
- Merge branch 'staging_distributed_cpu_fix' into 'staging' · 7890681a
  Mohammad Shoeybi authored Apr 14, 2020
```
Added logic for initilialize_megatron to work with cpu only jobs. This is...

See merge request ADLR/megatron-lm!46
```
  7890681a
- Merge branch 'staging_mp_merge' into 'staging_gpt2_dataloader' · e06159f2
  Jared Casper authored Apr 14, 2020
```
refactored merge_mp_partitions.py

See merge request ADLR/megatron-lm!47
```
  e06159f2
- Merge branch 'indexed_dataset_nocast' into 'staging_gpt2_dataloader' · 76e9a71e
  Mohammad Shoeybi authored Apr 13, 2020
```
Do not cast return type of indexed dataset

See merge request ADLR/megatron-lm!48
```
  76e9a71e
- added comment to build_index_mappings · d8bb57ad
  Mohammad authored Apr 13, 2020
  
  d8bb57ad
13 Apr, 2020 6 commits
- Do not cast data returned from indexed_dataset to int64, rely on caller to... · 71cf1866
  Jared Casper authored Apr 13, 2020
```
Do not cast data returned from indexed_dataset to int64, rely on caller to cast to appropriate type.
```
  71cf1866
- Add comment on purpose of test_indexed_dataset. · dff98d47
  Jared Casper authored Apr 13, 2020
  
  dff98d47
- refactored merge_mp_partitions.py · 03d0d93d
  Mohammad authored Apr 12, 2020
  
  03d0d93d
- cleaned up old gpt2 dataset stuff from openwebtext · 5655f076
  Mohammad authored Apr 12, 2020
  
  5655f076
- some cleanup: configure data, requirements, and docker · 20764e12
  Mohammad authored Apr 12, 2020
  
  20764e12
- data loading for BERT and GPT cleaned up · 2e38461b
  Mohammad authored Apr 12, 2020
  
  2e38461b
12 Apr, 2020 1 commit
- fixed a bug related to last index in the shuffled_idx · eb74fa34
  Mohammad authored Apr 12, 2020
  
  eb74fa34
11 Apr, 2020 2 commits
- Update initialize.py · 87bd026a
  Raul Puri authored Apr 10, 2020
  
  87bd026a
- Added logic for initilialize_megatron to work with cpu only jobs. This is... · 5448ca25
  Raul Puri authored Apr 10, 2020
```
Added logic for initilialize_megatron to work with cpu only jobs. This is necessary for several evaluation and processing scripts in downstream repos.
```
  5448ca25
10 Apr, 2020 2 commits
- fp32 fixes · 5f174c07
  Mohammad authored Apr 10, 2020
  
  5f174c07
- addressed Jareds comments · 1c1a55da
  Mohammad authored Apr 09, 2020
  
  1c1a55da
09 Apr, 2020 2 commits
- moved current gpt2 dataloader to old · 898fcb94
  Mohammad authored Apr 08, 2020
  
  898fcb94
- testing new gpt2 dataset · c3691006
  Mohammad authored Apr 08, 2020
  
  c3691006
08 Apr, 2020 8 commits
- Move old scripts to deprecated_data_utils. · 836c6776
  Jared Casper authored Apr 08, 2020
  
  836c6776
- Add sentence or document suffix to output filename of preprocess_data.py · 1e283449
  Jared Casper authored Apr 08, 2020
  
  1e283449
- Rename and organize scripts directory. · 985f027d
  Jared Casper authored Apr 08, 2020
  
  985f027d
- New tools directory. · 3500d036
  Jared Casper authored Apr 08, 2020
  
  3500d036
- Remove old preprocess_data.py · a6240676
  Jared Casper authored Apr 08, 2020
  
  a6240676
- Document indexed_dataset.get() a bit. · b1c4a2d6
  Jared Casper authored Apr 08, 2020
  
  b1c4a2d6
- Updates to preprocess_data.py and indexed_dataset. · da0562fc
  Jared Casper authored Apr 07, 2020
```
preprocess_data:
- Adds ability to not split sentences. This is used for gpt2 datasets.

- Adds ability to create multiple datasets from different json keys,
this is current untested.

indexed_dataset:
- Add new "get" function to get a portion of an entry.
```
  da0562fc
- added build sample index to c++ · f66c58a9
  Mohammad authored Apr 07, 2020
  
  f66c58a9
07 Apr, 2020 1 commit
- added new gpt2 dataloder · fe207787
  Mohammad authored Apr 06, 2020
  
  fe207787
03 Apr, 2020 9 commits
- Merge branch 'staging_ensemble' into 'staging' · 1363e503
  Mohammad Shoeybi authored Apr 03, 2020
```
added task ensembling

See merge request ADLR/megatron-lm!40
```
  1363e503
- functionalized code · ab754c8c
  Raul Puri authored Apr 03, 2020
  
  ab754c8c
- Merge branch 'dataset-utils-lint' into 'staging' · 1eccd462
  Mohammad Shoeybi authored Apr 03, 2020
```
Lint megatron/data/dataset_utils.py

See merge request ADLR/megatron-lm!42
```
  1eccd462
- Lint megatron/data/dataset_utils.py · 840759b8
  Neel Kant authored Apr 03, 2020
  
  840759b8
- added task ensembling · 5dd2a9ad
  Raul Puri authored Apr 03, 2020
  
  5dd2a9ad
- Merge branch 'staging_text_generation' into 'staging' · 63262827
  Raul Puri authored Apr 03, 2020
```
Refactoring text generation

See merge request ADLR/megatron-lm!39
```
  63262827
- sample generation runs · fffa0497
  Mohammad authored Apr 02, 2020
  
  fffa0497
- code runs · 752eeae3
  Mohammad authored Apr 02, 2020
  
  752eeae3
- generate samples linted · a6ba254f
  Mohammad authored Apr 02, 2020
  
  a6ba254f