Commit 06049edc authored by Rayyyyy's avatar Rayyyyy
Browse files

Update README

parent 82e6800e
......@@ -58,7 +58,7 @@ pip install -e .
## 训练
### xtuner微调方法
1. 训练库安装,请注意所需库版本
1. 训练库安装(非llama3_pytorch目录下),请注意所需库版本
```bash
pip uninstall flash-attn # 2.0.4+82379d7.abi0.dtk2404.torch2.1
# docker环境含有deepspeed的可不进行安装, 需要对照版本是否一致即可
......@@ -67,6 +67,8 @@ git clone -b v0.1.18 https://github.com/InternLM/xtuner.git
cd xtuner
pip install -e '.[all]'
pip install mmengine==0.10.3
# 注意bitsandbytes库版本,如果环境中一致可不安装,否则需要重新安装
pip install bitsandbytes-0.37.0+das1.0+gitd3d888f.abi0.dtk2404.torch2.1-py3-none-any.whl
```
2. 下载预训练模型,具体模型请修改`download_models.py`
```bash
......@@ -74,10 +76,10 @@ cd /your_code_path/llama3_pytorch
pip install modelscope
python download_models.py
```
2. 修改[llama3_8b_instruct_qlora_alpaca_e3_M.py](./llama3_8b_instruct_qlora_alpaca_e3_M.py)代码中的`pretrained_model_name_or_path``data_path`为本地模型、数据地址;
3. 根据硬件环境和自身训练需求来调整`max_length``batch_size``accumulative_counts``max_epochs``lr``save_steps``evaluation_freq`、model.lora中的`r``lora_alpha`参数,默认参数支持4*32G;
4. ${DCU_NUM}参数修改为要使用的DCU卡数量,不同数据集需要修改llama3_8b_instruct_qlora_alpaca_e3_M.py中`SYSTEM``evaluation_inputs``dataset_map_fn``train_dataloader.sampler``train_cfg`参数设置,详情请参考代码注释项,当前默认alpaca数据集,**`--work-dir`设定保存模型路径**
5. 执行
3. 修改[llama3_8b_instruct_qlora_alpaca_e3_M.py](./llama3_8b_instruct_qlora_alpaca_e3_M.py)代码中的`pretrained_model_name_or_path``data_path`为本地模型、数据地址;
4. 根据硬件环境和自身训练需求来调整`max_length``batch_size``accumulative_counts``max_epochs``lr``save_steps``evaluation_freq`、model.lora中的`r``lora_alpha`参数,默认参数支持4*32G;
5. ${DCU_NUM}参数修改为要使用的DCU卡数量,不同数据集需要修改llama3_8b_instruct_qlora_alpaca_e3_M.py中`SYSTEM``evaluation_inputs``dataset_map_fn``train_dataloader.sampler``train_cfg`参数设置,详情请参考代码注释项,当前默认alpaca数据集,**`--work-dir`设定保存模型路径**
6. 执行
```bash
bash finetune.sh
or
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment