Upload New File

01a42322 · wxj · cdf234ac · 01a42322
Commit 01a42322 authored Mar 17, 2025 by wxj
Hide whitespace changes
Inline Side-by-side

Showing with 8 additions and 0 deletions

qwen_data_process.sh qwen_data_process.sh +8 -0

No files found.
--- a/qwen_data_process.sh
+++ b/qwen_data_process.sh
+python tools/preprocess_data.py \
+    --input /public/home/wangxj/Downloads/datasets/oscar-1GB-head/oscar-1GB_head.jsonl \
+    --output-prefix /public/home/wangxj/Downloads/datasets/oscar-1GB-head/oscar-1GB_head-qwen \
+    --vocab-file /public/home/wangxj/Downloads/model_weights/qwen1.5_14b/vocab.json \
+    --tokenizer-type QwenTokenizer \
+    --merge-file /public/home/wangxj/Downloads/model_weights/qwen1.5_14b/merges.txt \
+    --append-eod \
+    --workers 8