README.md 10 KB
Newer Older
lvhan028's avatar
lvhan028 committed
1
<div align="center">
Lyu Han's avatar
Lyu Han committed
2
  <img src="resources/lmdeploy-logo.svg" width="450"/>
lvhan028's avatar
lvhan028 committed
3

RunningLeon's avatar
RunningLeon committed
4
5
6
7
8
9
10
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://lmdeploy.readthedocs.io/en/latest/)
[![badge](https://github.com/InternLM/lmdeploy/workflows/lint/badge.svg)](https://github.com/InternLM/lmdeploy/actions)
[![PyPI](https://img.shields.io/pypi/v/lmdeploy)](https://pypi.org/project/lmdeploy)
[![license](https://img.shields.io/github/license/InternLM/lmdeploy.svg)](https://github.com/InternLM/lmdeploy/tree/main/LICENSE)
[![issue resolution](https://img.shields.io/github/issues-closed-raw/InternLM/lmdeploy)](https://github.com/InternLM/lmdeploy/issues)
[![open issues](https://img.shields.io/github/issues-raw/InternLM/lmdeploy)](https://github.com/InternLM/lmdeploy/issues)

lvhan028's avatar
lvhan028 committed
11
12
13
14
English | [简体中文](README_zh-CN.md)

</div>

15
<p align="center">
vansin's avatar
vansin committed
16
    👋 join us on <a href="https://twitter.com/intern_lm" target="_blank">Twitter</a>, <a href="https://discord.gg/xa29JuW87d" target="_blank">Discord</a> and <a href="https://r.vansin.top/?r=internwx" target="_blank">WeChat</a>
17
</p>
lvhan028's avatar
lvhan028 committed
18

19
20
______________________________________________________________________

q.yao's avatar
q.yao committed
21
## News 🎉
22

Chen Xin's avatar
Chen Xin committed
23
- \[2023/09\] TurboMind supports Qwen-14B
Lyu Han's avatar
Lyu Han committed
24
- \[2023/09\] TurboMind supports InternLM-20B
Lyu Han's avatar
Lyu Han committed
25
- \[2023/09\] TurboMind supports all features of Code Llama: code completion, infilling, chat / instruct, and python specialist. Click [here](./docs/en/supported_models/codellama.md) for deployment guide
26
- \[2023/09\] TurboMind supports Baichuan2-7B
q.yao's avatar
q.yao committed
27
- \[2023/08\] TurboMind supports flash-attention2.
28
- \[2023/08\] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
Chen Xin's avatar
Chen Xin committed
29
- \[2023/08\] TurboMind supports Windows (tp=1)
30
- \[2023/08\] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation🚀. Check [this](./docs/en/w4a16.md) guide for detailed info
pppppM's avatar
pppppM committed
31
32
- \[2023/08\] LMDeploy has launched on the [HuggingFace Hub](https://huggingface.co/lmdeploy), providing ready-to-use 4-bit models.
- \[2023/08\] LMDeploy supports 4-bit quantization using the [AWQ](https://arxiv.org/abs/2306.00978) algorithm.
33
34
- \[2023/07\] TurboMind supports Llama-2 70B with GQA.
- \[2023/07\] TurboMind supports Llama-2 7B/13B.
q.yao's avatar
q.yao committed
35
- \[2023/07\] TurboMind supports tensor-parallel inference of InternLM.
36
37
38

______________________________________________________________________

lvhan028's avatar
lvhan028 committed
39
40
## Introduction

lvhan028's avatar
lvhan028 committed
41
42
LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the [MMRazor](https://github.com/open-mmlab/mmrazor) and [MMDeploy](https://github.com/open-mmlab/mmdeploy) teams. It has the following core features:

tpoisonooo's avatar
tpoisonooo committed
43
- **Efficient Inference Engine (TurboMind)**: Based on [FasterTransformer](https://github.com/NVIDIA/FasterTransformer), we have implemented an efficient inference engine - TurboMind, which supports the inference of LLaMA and its variant models on NVIDIA GPUs.
lvhan028's avatar
lvhan028 committed
44

45
- **Interactive Inference Mode**: By caching the k/v of attention during multi-round dialogue processes, it remembers dialogue history, thus avoiding repetitive processing of historical sessions.
lvhan028's avatar
lvhan028 committed
46

tpoisonooo's avatar
tpoisonooo committed
47
- **Multi-GPU Model Deployment and Quantization**: We provide comprehensive model deployment and quantification support, and have been validated at different scales.
48
49

- **Persistent Batch Inference**: Further optimization of model execution efficiency.
lvhan028's avatar
lvhan028 committed
50

pppppM's avatar
pppppM committed
51
![PersistentBatchInference](https://github.com/InternLM/lmdeploy/assets/67539920/e3876167-0671-44fc-ac52-5a0f9382493e)
lvhan028's avatar
lvhan028 committed
52

pppppM's avatar
pppppM committed
53
54
## Supported Models

55
`LMDeploy` has two inference backends, `Pytorch` and `TurboMind`. You can run `lmdeploy list` to check the supported model names.
pppppM's avatar
pppppM committed
56
57
58
59
60
61

### TurboMind

> **Note**<br />
> W4A16 inference requires Nvidia GPU with Ampere architecture or above.

Lyu Han's avatar
Lyu Han committed
62
63
64
65
|    Models    | Tensor Parallel | FP16 | KV INT8 | W4A16 | W8A8 |
| :----------: | :-------------: | :--: | :-----: | :---: | :--: |
|    Llama     |       Yes       | Yes  |   Yes   |  Yes  |  No  |
|    Llama2    |       Yes       | Yes  |   Yes   |  Yes  |  No  |
AllentDan's avatar
AllentDan committed
66
|    SOLAR     |       Yes       | Yes  |   Yes   |  Yes  |  No  |
Lyu Han's avatar
Lyu Han committed
67
68
| InternLM-7B  |       Yes       | Yes  |   Yes   |  Yes  |  No  |
| InternLM-20B |       Yes       | Yes  |   Yes   |  Yes  |  No  |
Lyu Han's avatar
Lyu Han committed
69
|   QWen-7B    |       Yes       | Yes  |   Yes   |  No   |  No  |
Chen Xin's avatar
Chen Xin committed
70
|   QWen-14B   |       Yes       | Yes  |   Yes   |  No   |  No  |
Lyu Han's avatar
Lyu Han committed
71
72
73
| Baichuan-7B  |       Yes       | Yes  |   Yes   |  Yes  |  No  |
| Baichuan2-7B |       Yes       | Yes  |   No    |  No   |  No  |
|  Code Llama  |       Yes       | Yes  |   No    |  No   |  No  |
pppppM's avatar
pppppM committed
74
75
76

### Pytorch

Lyu Han's avatar
Lyu Han committed
77
78
79
80
81
|   Models    | Tensor Parallel | FP16 | KV INT8 | W4A16 | W8A8 |
| :---------: | :-------------: | :--: | :-----: | :---: | :--: |
|    Llama    |       Yes       | Yes  |   No    |  No   |  No  |
|   Llama2    |       Yes       | Yes  |   No    |  No   |  No  |
| InternLM-7B |       Yes       | Yes  |   No    |  No   |  No  |
pppppM's avatar
pppppM committed
82

lvhan028's avatar
lvhan028 committed
83
84
## Performance

85
**Case I**: output token throughput with fixed input token and output token number (1, 2048)
lvhan028's avatar
lvhan028 committed
86

87
**Case II**: request throughput with real conversation data
lvhan028's avatar
lvhan028 committed
88

89
Test Setting: LLaMA-7B, NVIDIA A100(80G)
lvhan028's avatar
lvhan028 committed
90

91
92
The output token throughput of TurboMind exceeds 2000 tokens/s, which is about 5% - 15% higher than DeepSpeed overall and outperforms huggingface transformers by up to 2.3x.
And the request throughput of TurboMind is 30% higher than vLLM.
lvhan028's avatar
lvhan028 committed
93

94
![benchmark](https://github.com/InternLM/lmdeploy/assets/4560679/7775c518-608e-4e5b-be73-7645a444e774)
lvhan028's avatar
lvhan028 committed
95

lvhan028's avatar
lvhan028 committed
96
97
98
## Quick Start

### Installation
lvhan028's avatar
lvhan028 committed
99

100
Install lmdeploy with pip ( python 3.8+) or [from source](./docs/en/build.md)
lvhan028's avatar
lvhan028 committed
101
102

```shell
lvhan028's avatar
lvhan028 committed
103
pip install lmdeploy
lvhan028's avatar
lvhan028 committed
104
105
```

106
107
108
109
110
111
112
113
> **Note**<br />
> `pip install lmdeploy` can only install the runtime required packages. If users want to run codes from modules like `lmdeploy.lite` and `lmdeploy.serve`, they need to install the extra required packages.
> For instance, running `pip install lmdeploy[lite]` would install extra dependencies for `lmdeploy.lite` module.
>
> - `all`: Install lmdeploy with all dependencies in `requirements.txt`
> - `lite`: Install lmdeploy with extra dependencies in `requirements/lite.txt`
> - `serve`: Install lmdeploy with dependencies in `requirements/serve.txt`

lvhan028's avatar
lvhan028 committed
114
### Deploy InternLM
lvhan028's avatar
lvhan028 committed
115

lvhan028's avatar
lvhan028 committed
116
#### Get InternLM model
lvhan028's avatar
lvhan028 committed
117
118

```shell
lvhan028's avatar
lvhan028 committed
119
# 1. Download InternLM model
lvhan028's avatar
lvhan028 committed
120

pppppM's avatar
pppppM committed
121
122
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
123
git clone https://huggingface.co/internlm/internlm-chat-7b-v1_1 /path/to/internlm-chat-7b
pppppM's avatar
pppppM committed
124
125
126
127
128

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1

lvhan028's avatar
lvhan028 committed
129
# 2. Convert InternLM model to turbomind's format, which will be in "./workspace" by default
130
lmdeploy convert internlm-chat-7b /path/to/internlm-chat-7b
lvhan028's avatar
lvhan028 committed
131
132
133

```

lvhan028's avatar
lvhan028 committed
134
#### Inference by TurboMind
lvhan028's avatar
lvhan028 committed
135
136

```shell
137
lmdeploy chat turbomind ./workspace
lvhan028's avatar
lvhan028 committed
138
139
```

140
141
142
143
144
145
146
> **Note**<br />
> When inferring with FP16 precision, the InternLM-7B model requires at least 15.7G of GPU memory overhead on TurboMind. <br />
> It is recommended to use NVIDIA cards such as 3090, V100, A100, etc.
> Disable GPU ECC can free up 10% memory, try `sudo nvidia-smi --ecc-config=0` and reboot system.

> **Note**<br />
> Tensor parallel is available to perform inference on multiple GPUs. Add `--tp=<num_gpu>` on `chat` to enable runtime TP.
lvhan028's avatar
lvhan028 committed
147

148
149
150
#### Serving with gradio

```shell
151
152
153
# install lmdeploy with extra dependencies
pip install lmdeploy[serve]

154
lmdeploy serve gradio ./workspace
155
156
157
158
```

![](https://github.com/InternLM/lmdeploy/assets/67539920/08d1e6f2-3767-44d5-8654-c85767cec2ab)

159
160
161
162
163
#### Serving with Restful API

Launch inference server by:

```shell
164
165
166
# install lmdeploy with extra dependencies
pip install lmdeploy[serve]

167
lmdeploy serve api_server ./workspace --instance_num 32 --tp 1
168
169
170
171
172
173
```

Then, you can communicate with it by command line,

```shell
# restful_api_url is what printed in api_server.py, e.g. http://localhost:23333
174
lmdeploy serve api_client api_server_url
175
176
177
178
179
```

or webui,

```shell
180
# api_server_url is what printed in api_server.py, e.g. http://localhost:23333
181
# server_ip and server_port here are for gradio ui
182
183
# example: lmdeploy serve gradio http://localhost:23333 --server_name localhost --server_port 6006
lmdeploy serve gradio api_server_url --server_name ${gradio_ui_ip} --server_port ${gradio_ui_port}
184
185
186
187
```

Refer to [restful_api.md](docs/en/restful_api.md) for more details.

188
#### Serving with Triton Inference Server
lvhan028's avatar
lvhan028 committed
189

lvhan028's avatar
lvhan028 committed
190
Launch inference server by:
lvhan028's avatar
lvhan028 committed
191
192

```shell
lvhan028's avatar
lvhan028 committed
193
bash workspace/service_docker_up.sh
lvhan028's avatar
lvhan028 committed
194
195
```

lvhan028's avatar
lvhan028 committed
196
Then, you can communicate with the inference server by command line,
lvhan028's avatar
lvhan028 committed
197
198

```shell
199
python3 -m pip install tritonclient[grpc]
200
lmdeploy serve triton_client {server_ip_addresss}:33337
lvhan028's avatar
lvhan028 committed
201
202
```

lvhan028's avatar
lvhan028 committed
203
or webui,
AllentDan's avatar
AllentDan committed
204

vansin's avatar
vansin committed
205
```shell
206
lmdeploy serve gradio {server_ip_addresss}:33337
AllentDan's avatar
AllentDan committed
207
208
```

209
For the deployment of other supported models, such as LLaMA, LLaMA-2, vicuna and so on, you can find the guide from [here](docs/en/serving.md)
lvhan028's avatar
lvhan028 committed
210

WRH's avatar
WRH committed
211
212
### Inference with PyTorch

213
For detailed instructions on Inference pytorch models, see [here](docs/en/pytorch.md).
214

WRH's avatar
WRH committed
215
216
217
#### Single GPU

```shell
218
lmdeploy chat torch $NAME_OR_PATH_TO_HF_MODEL \
WRH's avatar
WRH committed
219
220
221
222
223
224
225
226
227
    --max_new_tokens 64 \
    --temperture 0.8 \
    --top_p 0.95 \
    --seed 0
```

#### Tensor Parallel with DeepSpeed

```shell
WRH's avatar
WRH committed
228
deepspeed --module --num_gpus 2 lmdeploy.pytorch.chat \
WRH's avatar
WRH committed
229
230
231
232
233
234
235
    $NAME_OR_PATH_TO_HF_MODEL \
    --max_new_tokens 64 \
    --temperture 0.8 \
    --top_p 0.95 \
    --seed 0
```

236
237
238
239
240
241
You need to install deepspeed first to use this feature.

```
pip install deepspeed
```

242
243
## Quantization

pppppM's avatar
pppppM committed
244
245
#### Weight INT4 Quantization

246
LMDeploy uses [AWQ](https://arxiv.org/abs/2306.00978) algorithm for model weight quantization
pppppM's avatar
pppppM committed
247

248
[Click here](./docs/en/w4a16.md) to view the test results for weight int4 usage.
249

250
#### KV Cache INT8 Quantization
lvhan028's avatar
lvhan028 committed
251

252
[Click here](./docs/en/kv_int8.md) to view the usage method, implementation formula, and test results for kv int8.
253

254
> **Warning**<br />
255
> runtime Tensor Parallel for quantized model is not available. Please setup `--tp` on `deploy` to enable static TP.
256

lvhan028's avatar
lvhan028 committed
257
## Contributing
lvhan028's avatar
lvhan028 committed
258

lvhan028's avatar
lvhan028 committed
259
We appreciate all contributions to LMDeploy. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.
260

lvhan028's avatar
lvhan028 committed
261
262
263
## Acknowledgement

- [FasterTransformer](https://github.com/NVIDIA/FasterTransformer)
pppppM's avatar
pppppM committed
264
- [llm-awq](https://github.com/mit-han-lab/llm-awq)
lvhan028's avatar
lvhan028 committed
265
266
267
268

## License

This project is released under the [Apache 2.0 license](LICENSE).