Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
qwen_data_process.sh 463 Bytes
Newer Older
wxj's avatar
Upload New File  
wxj committed
1
2
3
4
5
6
7
8 
python tools/preprocess_data.py \
    --input /public/home/wangxj/Downloads/datasets/oscar-1GB-head/oscar-1GB_head.jsonl \
    --output-prefix /public/home/wangxj/Downloads/datasets/oscar-1GB-head/oscar-1GB_head-qwen \
    --vocab-file /public/home/wangxj/Downloads/model_weights/qwen1.5_14b/vocab.json \
    --tokenizer-type QwenTokenizer \
    --merge-file /public/home/wangxj/Downloads/model_weights/qwen1.5_14b/merges.txt \
    --append-eod \
    --workers 8