Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
LLaMA_TencentPretrain_pytorch
Commits
95e21a4a
Commit
95e21a4a
authored
Oct 20, 2023
by
zhaoying1
Browse files
Update README.md
parent
884640da
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
2 deletions
+2
-2
README.md
README.md
+2
-2
No files found.
README.md
View file @
95e21a4a
...
...
@@ -92,7 +92,7 @@ $ tree ./data/
── dataset.pt
```
## 模型权重下载
##
#
模型权重下载
1.
方式一:下载huggingface格式模型。以 7B 模型为例,首先下载预训练
[
LLaMA权重
](
https://huggingface.co/decapoda-research/llama-7b-hf
)
,转换到TencentPretrain格式:
```
commandline
python3 scripts/convert_llama_from_huggingface_to_tencentpretrain.py --input_model_path $LLaMA_HF_PATH \
...
...
@@ -183,7 +183,7 @@ cd multi_node
bash run-13b.sh
```
## 模型分块
##
#
模型分块
训练初始化时,每张卡会加载一个模型的拷贝,因此内存需求为模型大小
*
GPU数量。内存不足时可以通过以下方式将模型分块,然后使用分块加载。
```
commandline
python3 scripts/convert_model_into_blocks.py \
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment