Commits · 69d8ff1cf4255ca2c628c366676e4c443d9264bb · OpenDAS / Megatron-LM

"vscode:/vscode.git/clone" did not exist on "4673b96f9298bc6e8ad25adbde0bfd17d57ff232"

13 Apr, 2020 1 commit
- refactored merge_mp_partitions.py · 03d0d93d
  Mohammad authored Apr 12, 2020
  
  03d0d93d
08 Apr, 2020 3 commits
- Add sentence or document suffix to output filename of preprocess_data.py · 1e283449
  Jared Casper authored Apr 08, 2020
  
  1e283449
- New tools directory. · 3500d036
  Jared Casper authored Apr 08, 2020
  
  3500d036
- Updates to preprocess_data.py and indexed_dataset. · da0562fc
  Jared Casper authored Apr 07, 2020
```
preprocess_data:
- Adds ability to not split sentences. This is used for gpt2 datasets.

- Adds ability to create multiple datasets from different json keys,
this is current untested.

indexed_dataset:
- Add new "get" function to get a portion of an entry.
```
  da0562fc