Commits · a14f1442f791de2e49ca0799acb6dddb68ca77a4 · OpenDAS / Megatron-LM

14 Apr, 2020 1 commit
- Modify preprocess_data.py to accommodate titles dataset · 27ecc17a
  Neel Kant authored Apr 13, 2020
  
  27ecc17a
08 Apr, 2020 1 commit
- Remove old preprocess_data.py · a6240676
  Jared Casper authored Apr 08, 2020
  
  a6240676
05 Apr, 2020 1 commit
- Hacks to build IndexedDataset and run pretrain · b03af49e
  Neel Kant authored Apr 05, 2020
  
  b03af49e
09 Nov, 2019 1 commit
- Skip any empty sentences during preprocessing. · 3f4bc91b
  Jared Casper authored Nov 08, 2019
  
  3f4bc91b
08 Nov, 2019 1 commit
- Add document index to index file. An empty sentence no longer separate documents. · 87bbe9be
  Jared Casper authored Nov 07, 2019
  
  87bbe9be
07 Nov, 2019 2 commits
- Initial commit of multiprocess preprocess and extracted copy of fairseq's indexed_dataset. · 1237533e
  Jared Casper authored Nov 07, 2019
  
  1237533e
- added bert tokenization · 0ceeb3b4
  Mohammad Shoeybi authored Nov 06, 2019
  
  0ceeb3b4