README.md 8.77 KB
Newer Older
xiabo's avatar
xiabo committed
1
2
3
4
# <div align="center"><strong>LMdeploy</strong></div>
## 简介
LMDeploy 由 [MMDeploy](https://github.com/open-mmlab/mmdeploy)[MMRazor](https://github.com/open-mmlab/mmrazor) 团队联合开发,是涵盖了 LLM 任务的全套轻量化、部署和服务解决方案。
这个强大的工具箱提供以下核心功能:
lvhan028's avatar
lvhan028 committed
5

xiabo's avatar
xiabo committed
6
- **高效推理引擎 TurboMind**:基于 [FasterTransformer](https://github.com/NVIDIA/FasterTransformer),我们实现了高效推理引擎 TurboMind,支持 InternLM、LLaMA、vicuna等模型在 NVIDIA GPU 上的推理。
RunningLeon's avatar
RunningLeon committed
7

xiabo's avatar
xiabo committed
8
- **交互推理方式**:通过缓存多轮对话过程中 attention 的 k/v,记住对话历史,从而避免重复处理历史会话。
lvhan028's avatar
lvhan028 committed
9

xiabo's avatar
xiabo committed
10
- **多 GPU 部署和量化**:我们提供了全面的模型部署和量化支持,已在不同规模上完成验证。
lvhan028's avatar
lvhan028 committed
11

xiabo's avatar
xiabo committed
12
- **persistent batch 推理**:进一步优化模型执行效率。
lvhan028's avatar
lvhan028 committed
13

xiabo's avatar
xiabo committed
14
15
persistent batch 推理:进一步优化模型执行效率。
LMdeploy官方github地址:[https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy)
xiabo's avatar
xiabo committed
16
17
18
19
20
21
22
23
24
25
26
## 支持模型
|     模型     | 模型并行 | FP16 | KV INT8 |
| :----------: | :------: | :--: | :-----: |
|    Llama     |   Yes    | Yes  |   Yes   |
|    Llama2    |   Yes    | Yes  |   Yes   |
| InternLM-7B  |   Yes    | Yes  |   Yes   |
| InternLM-20B |   Yes    | Yes  |   Yes   |
|   QWen-7B    |   Yes    | Yes  |   Yes   |
|   QWen-14B   |   Yes    | Yes  |   Yes   |
| Baichuan-7B  |   Yes    | Yes  |   Yes   |
| Baichuan2-7B |   Yes    | Yes  |   No    |
27

xiabo's avatar
xiabo committed
28
## 安装
29

xiabo's avatar
xiabo committed
30
### 使用源码编译方式安装
31

xiabo's avatar
xiabo committed
32
#### 编译环境准备
xiabo's avatar
xiabo committed
33
34
下载光源的镜像,起dcoker
```
xiabo's avatar
xiabo committed
35
docker pull image.sourcefind.cn:5000/dcu/admin/base/custom:lmdeploy_dtk2310_torch1.13_py38
xiabo's avatar
xiabo committed
36

xiabo's avatar
xiabo committed
37
38
39
40
# <Image ID>用上面拉取docker镜像的ID替换
# <Host Path>主机端路径
# <Container Path>容器映射路径
docker run -it --name baichuan --shm-size=1024G  --device=/dev/kfd --device=/dev/dri/ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 --ipc=host --network host --group-add video -v <Host Path>:<Container Path> <Image ID> /bin/bash
xiabo's avatar
xiabo committed
41
```
xiabo's avatar
xiabo committed
42
注:要是非光源提供镜像,配置环境:(若安装过慢,可以添加源:pip3 install xxx -i  https://pypi.tuna.tsinghua.edu.cn/simple/)
xiabo's avatar
xiabo committed
43
```
xiabo's avatar
xiabo committed
44
45
pip3 install -r requirements.txt
pip3 install urllib3==1.24
xiabo's avatar
xiabo committed
46
yum install rapidjson
xiabo's avatar
xiabo committed
47
48
49
50

# 执行dtk环境变量
source {DTK_PATH}/env.sh
source {DTK_PATH}/cuda/env.sh
xiabo's avatar
xiabo committed
51
52
53
54
55
export GPU_MAX_HW_QUEUES=4
export HIP_DIRECT_DISPATCH=0
export GPU_FLUSH_ON_EXECUTION=1
export AMD_SERIALIZE_KERNEL=1
export AMD_SERIALIZE_COPY=1
56
57
```

xiabo's avatar
xiabo committed
58
59
#### 源码编译安装
- 代码下载
xiabo's avatar
xiabo committed
60
61
根据不同的需求下载不同的分支
- 提供2种源码编译方式(进入lmdeploy目录):
62
```
xiabo's avatar
xiabo committed
63
64
65
66
67
68
69
1. 源码编译安装
mkdir build && cd build
sh ../generate.sh
make -j 32 && make install
cd .. && python3 setup.py install

2. 编译成whl包安装
xiabo's avatar
xiabo committed
70
71
# 安装wheel 
pip3 install wheel 
xiabo's avatar
xiabo committed
72
73
74
75
76
mkdir build && cd build
sh ../generate.sh
make -j 32 && make install
cd .. && python3 setup.py bdist_wheel
cd dist && pip3 install lmdeploy*
lvhan028's avatar
lvhan028 committed
77
```
xiabo's avatar
xiabo committed
78

xiabo's avatar
xiabo committed
79
## 模型服务
lvhan028's avatar
lvhan028 committed
80

xiabo's avatar
xiabo committed
81
82
83
84
85
86
87
88
89
90
91
92
### 部署 [LLaMA](https://huggingface.co/huggyllama) 服务
请从[这里](https://huggingface.co/huggyllama) 下载 llama 模型,参考如下命令部署服务:
以7B为例:
```
1、模型转换
# <model_name> 模型的名字 ('llama', 'internlm', 'vicuna', 'internlm-chat-7b', 'internlm-chat', 'internlm-chat-7b-8k', 'internlm-chat-20b', 'internlm-20b', 'baichuan-7b', 'baichuan2-7b', 'llama2', 'qwen-7b', 'qwen-14b',)
# <model_path> 模型路径
# <model_format> 模型的格式 ('llama', 'hf', 'qwen')
# <tokenizer_path> tokenizer模型的路径(默认None,会去model_path里面找qwen.tiktoken)
# <model_format> 保存输出的目标路径(默认./workspace)
# <tp> 用于张量并行的GPU数量应该是2^n

xiabo's avatar
xiabo committed
93
lmdeploy convert --model_name llama --model_path /path/to/model --model_format hf --tokenizer_path None --dst_path ./workspace_llama --tp 1
xiabo's avatar
xiabo committed
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

2、运行
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_llama --tp 1     # 输入问题后执行2次回车进行推理

# 在服务器界面运行:
在bash端运行:
# <model_path_or_server> 部署模型的路径或tritonserver URL或restful api URL。前者用于与gradio直接运行服务。后者用于默认情况下使用tritonserver运行。如果输入URL是restful api。请启用另一个标志“restful_api”。
# <server_name> gradio服务器的ip地址
# <server_port> gradio服务器的ip的端口
# <batch_size> 于直接运行Turbomind的batch大小 (默认32)
# <tp> 用于张量并行的GPU数量应该是2^n (和模型转换的时候保持一致)
# <restful_api> modelpath_or_server的标志(默认是False)

lmdeploy serve gradio --model_path_or_server ./workspace_llama --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话

```
### 部署 [llama2](https://huggingface.co/meta-llama) 服务
xiabo's avatar
xiabo committed
114
115
请从[这里](https://huggingface.co/meta-llama) 下载 llama2 模型,参考如下命令部署服务:
以7B为例:
lvhan028's avatar
lvhan028 committed
116
```
xiabo's avatar
xiabo committed
117
1、模型转换
xiabo's avatar
xiabo committed
118
lmdeploy convert --model_name llama2 --model_path /path/to/model --model_format hf --tokenizer_path None --dst_path ./workspace_llama2 --tp 1  # 
xiabo's avatar
xiabo committed
119
2、运行
xiabo's avatar
xiabo committed
120
121
122
123
124
125
126
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_llama2 --tp 1
# 在服务器界面运行:
在bash端运行:
lmdeploy serve gradio --model_path_or_server ./workspace_llama2 --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话
AllentDan's avatar
AllentDan committed
127
```
xiabo's avatar
xiabo committed
128
### 部署 [internlm](https://huggingface.co/internlm/) 服务
xiabo's avatar
xiabo committed
129
请从[这里](https://huggingface.co/internlm) 下载 internlm 模型,参考如下命令部署服务:
xiabo's avatar
xiabo committed
130
以7B为例:
131
```
xiabo's avatar
xiabo committed
132
1、模型转换
xiabo's avatar
xiabo committed
133
lmdeploy convert --model_name model_name --model_path /path/to/model --model_format hf --tokenizer_path None --dst_path ./workspace_intern --tp 1  # 根据模型的类型选择model_name是internlm-chat还是internlm
xiabo's avatar
xiabo committed
134
2、运行
xiabo's avatar
xiabo committed
135
136
137
138
139
140
141
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_intern --tp 1
# 在服务器界面运行:
在bash端运行:
lmdeploy serve gradio --model_path_or_server ./workspace_intern --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话
142
```
xiabo's avatar
xiabo committed
143
144
145
146
147
### 部署 [baichuan](https://huggingface.co/baichuan-inc) 服务
请从[这里](https://huggingface.co/baichuan-inc) 下载 baichuan 模型,参考如下命令部署服务:
以7B为例:
```
1、模型转换
xiabo's avatar
xiabo committed
148
lmdeploy convert --model_name baichuan-7b --model_path /path/to/model --model_format hf --tokenizer_path None --dst_path ./workspace_baichuan --tp 1
xiabo's avatar
xiabo committed
149
2、运行
xiabo's avatar
xiabo committed
150
151
152
153
154
155
156
157
158
159
160
161
162
163
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_baichuan --tp 1
# 在服务器界面运行:
在bash端运行:
lmdeploy serve gradio --model_path_or_server ./workspace_baichuan --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话
```

### 部署 [baichuan2](https://huggingface.co/baichuan-inc) 服务
请从[这里](https://huggingface.co/baichuan-inc) 下载 baichuan2 模型,参考如下命令部署服务:
以7B为例:
```
1、模型转换
xiabo's avatar
xiabo committed
164
lmdeploy convert --model_name baichuan2-7b --model_path /path/to/model --model_format hf --tokenizer_path None --dst_path ./workspace_baichuan2 --tp 1
xiabo's avatar
xiabo committed
165
166
167
168
169
170
171
172
2、运行
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_baichuan2 --tp 1
# 在服务器界面运行:
在bash端运行:
lmdeploy serve gradio --model_path_or_server ./workspace_baichuan2 --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话
xiabo's avatar
xiabo committed
173
```
xiabo's avatar
xiabo committed
174

xiabo's avatar
xiabo committed
175
176
177
178
179
### 部署 [qwen](https://huggingface.co/Qwen) 服务
请从[这里](https://huggingface.co/Qwen) 下载 qwen 模型,参考如下命令部署服务:
以7B为例:
```
1、模型转换
xiabo's avatar
xiabo committed
180
lmdeploy convert --model_name qwen-7b --model_path /path/to/model --model_format qwen --tokenizer_path None --dst_path ./workspace_qwen --tp 1
xiabo's avatar
xiabo committed
181
2、运行
xiabo's avatar
xiabo committed
182
183
184
185
186
187
188
# bash界面运行
lmdeploy chat turbomind --model_path ./workspace_qwen --tp 1
# 在服务器界面运行:
在bash端运行:
lmdeploy serve gradio --model_path_or_server ./workspace_qwen --server_name {ip} --server_port {pord} --batch_size 32 --tp 1 --restful_api False 

在网页上输入{ip}:{pord}即可进行对话
xiabo's avatar
xiabo committed
189
```
xiabo's avatar
xiabo committed
190
191
192
193

## result
![qwen推理](docs/dcu/qwen推理.gif)

xiabo's avatar
xiabo committed
194
195
196
### 详细可参考 [docs](./docs/zh_cn/serving.md) 
## 版本号查询
- python -c "import lmdeploy; lmdeploy.\_\_version__",版本号与官方版本同步,查询该软件的版本号,例如0.0.6;
197

xiabo's avatar
xiabo committed
198
199
## Known Issue
-
lvhan028's avatar
lvhan028 committed
200

xiabo's avatar
xiabo committed
201
202
## Note
+ 若使用pip install下载安装过慢,可添加pypi清华源:-i https://pypi.tuna.tsinghua.edu.cn/simple/
lvhan028's avatar
lvhan028 committed
203

xiabo's avatar
xiabo committed
204
205
206
## 其他参考
- [README_origin](README_origin.md)
- [README_zh-CN](README_zh-CN.md)