Skip to content

GitLab

"awq/vscode:/vscode.git/clone" did not exist on "ce4a6bb1c238c014a06672cb74f6865573494d66"
Switch branch/tag
Find file Normal viewHistoryPermalink
test_preprocess_data.sh 241 Bytes
Newer Older
Jared Casper's avatar
Initial commit of multiprocess preprocess and extracted copy of fairseq's indexed_dataset.  
Jared Casper committed
1
2 
#!/bin/bash
Jared Casper's avatar
Add document index to index file. An empty sentence no longer separate documents.  
Jared Casper committed
3 
IMPL=cached
Jared Casper's avatar
Initial commit of multiprocess preprocess and extracted copy of fairseq's indexed_dataset.  
Jared Casper committed
4
5
6 
python ../preprocess_data.py \
       --input test_samples.json \
       --vocab vocab.txt \
Jared Casper's avatar
Add document index to index file. An empty sentence no longer separate documents.  
Jared Casper committed
7
8 
       --dataset-impl ${IMPL} \
       --output-prefix test_samples_${IMPL} \
Jared Casper's avatar
Initial commit of multiprocess preprocess and extracted copy of fairseq's indexed_dataset.  
Jared Casper committed
9
10 
       --workers 1 \
       --log-interval 2