Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
LLaMA_Fastchat_pytorch
Commits
35cc1481
Commit
35cc1481
authored
Aug 13, 2024
by
dcuai
Browse files
Update README.md
parent
b9392018
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
5 deletions
+5
-5
README.md
README.md
+5
-5
No files found.
README.md
View file @
35cc1481
...
@@ -118,7 +118,7 @@ pip3 uninstall wandb
...
@@ -118,7 +118,7 @@ pip3 uninstall wandb
7B:
[
aimodels/llama-7b-hf
](
http://113.200.138.88:18080/aimodels/llama-7b-hf
)
7B:
[
aimodels/llama-7b-hf
](
http://113.200.138.88:18080/aimodels/llama-7b-hf
)
该训练脚本需要2节点,每节点8张DCU-Z100L-32G。
按需更改mpi_single.sh中模型权重所在路径。
按需更改mpi_single.sh中模型权重所在路径。
并行配置采用zero3,使用fp16精度微调,如果想使能apex adamw_apex_fused优化器,更改./FastChat-main/fastchat/train/train.py:55行优化器改成adamw_apex_fused。deepspeed config.json如下:
并行配置采用zero3,使用fp16精度微调,如果想使能apex adamw_apex_fused优化器,更改./FastChat-main/fastchat/train/train.py:55行优化器改成adamw_apex_fused。deepspeed config.json如下:
...
@@ -147,12 +147,12 @@ pip3 uninstall wandb
...
@@ -147,12 +147,12 @@ pip3 uninstall wandb
}
}
}
}
```
```
<!--该训练脚本需要2节点,每节点8张DCU-Z100L-32G。
进入节点1,根据环境修改hostfile,保证两节点文件路径一致,配置相同,按需修改mpi_job.sh中--mca btl_tcp_if_include enp97s0f1,enp97s0f1改为ip a命令后对应节点ip的网卡名,numa可以根据当前节点拓扑更改绑定,微调命令:
进入节点1,根据环境修改hostfile,保证两节点文件路径一致,配置相同,按需修改mpi_job.sh中--mca btl_tcp_if_include enp97s0f1,enp97s0f1改为ip a命令后对应节点ip的网卡名,numa可以根据当前节点拓扑更改绑定,微调命令:
-->
```
```
#注释mpi_
job.sh、mpi_
single.sh中的source env.sh
#注释mpi_single.sh中的source env.sh
,根据环境修改hostfile
bash mpi_job
.sh
mpirun -np 8 --allow-run-as-root --hostfile hostfile --bind-to none mpi_single
.sh
8
```
```
如果单节点运行7B的模型出现oom,可以适当减少batch size。
如果单节点运行7B的模型出现oom,可以适当减少batch size。
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment