Skip to content

GitLab

Commit 84fe5a1c authored by yangzhong's avatar yangzhong
Browse files

update README

parent f59c7272
Pipeline #3021 failed with stages
in 0 seconds
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
README.md
View file @ 84fe5a1c
...@@ -38,6 +38,8 @@ python tools/preprocess_data.py \ ...@@ -38,6 +38,8 @@ python tools/preprocess_data.py \
--workers 16 \ --workers 16 \
--append-eod --append-eod
``` ```
输出openwebtext-llama-7b_text_document.bin openwebtext-llama-7b_text_document.idx
.bin文件31G
### 下载tokenizer文件 ### 下载tokenizer文件
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment