README.md 13.4 KB
Newer Older
Casper's avatar
Casper committed
1
# AutoAWQ
Ji Lin's avatar
Ji Lin committed
2

Casper's avatar
Casper committed
3
4
<p align="center">
| <a href="https://github.com/casper-hansen/AutoAWQ/issues/32"><b>Roadmap</b></a> | <a href="https://github.com/casper-hansen/AutoAWQ/tree/main/examples"><b>Examples</b></a> | <a href="https://github.com/casper-hansen/AutoAWQ/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22"><b>Issues: Help Wanted</b></a> |
Casper's avatar
Casper committed
5
6
7
8

</p>
<p align="center">
    <a href="https://huggingface.co/models?search=awq">
Casper's avatar
Casper committed
9
        <img alt="Huggingface - Models" src="https://img.shields.io/badge/🤗_1000+_models_available-8A2BE2">
Casper's avatar
Casper committed
10
11
12
13
14
    </a>
    <a href="https://github.com/casper-hansen/AutoAWQ/releases">
        <img alt="GitHub - Releases" src="https://img.shields.io/github/release/casper-hansen/AutoAWQ.svg">
    </a>
    <a href="https://pypi.org/project/autoawq/">
Casper's avatar
Casper committed
15
        <img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/autoawq/month">
Casper's avatar
Casper committed
16
    </a>
Casper's avatar
Casper committed
17
</p>
Ji Lin's avatar
Ji Lin committed
18

Casper's avatar
Casper committed
19
AutoAWQ is an easy-to-use package for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs.  AutoAWQ was created and improved upon from the [original work](https://github.com/mit-han-lab/llm-awq) from MIT.
Ji Lin's avatar
Ji Lin committed
20

Casper's avatar
Casper committed
21
*Latest News* 🔥
Casper's avatar
Casper committed
22
- [2023/11] AutoAWQ inference has been integrated into 🤗 transformers. Now includes CUDA 12.1 wheels.
Casper's avatar
Casper committed
23
- [2023/10] Mistral (Fused Modules), Bigcode, Turing support, Memory Bug Fix (Saves 2GB VRAM)
Casper Hansen's avatar
Casper Hansen committed
24
- [2023/09] 1.6x-2.5x speed boost on fused models (now including MPT and Falcon).
Casper's avatar
Casper committed
25
26
- [2023/09] Multi-GPU support, bug fixes, and better benchmark scripts available
- [2023/08] PyPi package released and AutoModel class available
Ji Lin's avatar
Ji Lin committed
27
28
29

## Install

Casper's avatar
Casper committed
30
### Prerequisites
Casper's avatar
Casper committed
31

Casper's avatar
Casper committed
32
33
- Your GPU(s) must be of Compute Capability 7.5. Turing and later architectures are supported.
- Your CUDA version must be CUDA 11.8 or later.
Casper's avatar
Casper committed
34

Casper's avatar
Casper committed
35
### Install from PyPi
Casper's avatar
Casper committed
36

Casper's avatar
Casper committed
37
To install the newest AutoAWQ from PyPi, you need CUDA 12.1 installed.
Casper's avatar
Casper committed
38
39

```
Casper's avatar
Casper committed
40
pip install autoawq
Casper's avatar
Casper committed
41
42
```

Casper's avatar
Casper committed
43
If you cannot use CUDA 12.1, you can still use CUDA 11.8 and install the wheel from the [latest release](https://github.com/casper-hansen/AutoAWQ/releases).
Casper's avatar
Casper committed
44
45
46
47
48

```
pip install https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl
```

Casper's avatar
Casper committed
49
### Build from source
Casper's avatar
Casper committed
50

Casper's avatar
Casper committed
51
Build time can take 10-20 minutes. Download your model while you install AutoAWQ.
Casper's avatar
Casper committed
52

Ji Lin's avatar
Ji Lin committed
53
```
Casper's avatar
Casper committed
54
git clone https://github.com/casper-hansen/AutoAWQ
Casper's avatar
Casper committed
55
cd AutoAWQ
Ji Lin's avatar
Ji Lin committed
56
57
58
pip install -e .
```

Casper's avatar
Casper committed
59
## Supported models
Casper Hansen's avatar
Casper Hansen committed
60

Casper's avatar
Casper committed
61
The detailed support list:
Haotian (Ken) Tang's avatar
Haotian (Ken) Tang committed
62

Casper's avatar
Casper committed
63
64
65
66
| Models   | Sizes                       |
| ---------| ----------------------------|
| LLaMA-2  | 7B/13B/70B                  |
| LLaMA    | 7B/13B/30B/65B              |
Casper's avatar
Casper committed
67
| Mistral  | 7B                          |
Casper's avatar
Casper committed
68
69
70
71
72
| Vicuna   | 7B/13B                      |
| MPT      | 7B/30B                      |
| Falcon   | 7B/40B                      |
| OPT      | 125m/1.3B/2.7B/6.7B/13B/30B |
| Bloom    | 560m/3B/7B/                 |
Casper's avatar
Casper committed
73
| GPTJ     | 6.7B                        |
ldwang's avatar
ldwang committed
74
75
| Aquila   | 7B                          |
| Aquila2  | 7B/34B                      |
Casper's avatar
Casper committed
76
77
78
79
80
| Yi       | 6B/34B                      |
| Qwen     | 1.8B/7B/14B/72B             |
| BigCode  | 1B/7B/15B                   |
| GPT NeoX | 20B                         |
| GPT-J    | 6B                          |
Casper's avatar
Casper committed
81
82

## Usage
Ji Lin's avatar
Ji Lin committed
83

Casper's avatar
Casper committed
84
85
Under examples, you can find examples of how to quantize, run inference, and benchmark AutoAWQ models.

86
87
### INT4 GEMM vs INT4 GEMV vs FP16

Casper's avatar
Casper committed
88
There are two versions of AWQ: GEMM and GEMV. Both names relate to how matrix multiplication runs under the hood. We suggest the following:
89

Casper's avatar
Casper committed
90
- GEMV (quantized): 20% faster than GEMM, only batch size 1 (not good for large context).
Casper's avatar
Casper committed
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
- GEMM (quantized): Much faster than FP16 at batch sizes below 8 (good with large contexts).
- FP16 (non-quantized): Recommended for highest throughput: [vLLM](https://github.com/vllm-project/vllm).

#### Compute-bound vs Memory-bound

At small batch sizes with small 7B models, we are memory-bound. This means we are bound by the bandwidth our GPU has to push around the weights in memory, and this is essentially what limits how many tokens per second we can generate. Being memory-bound is what makes quantized models faster because your weights are 3x smaller and can therefore be pushed around in memory much faster. This is different from being compute-bound where the main time spent during generation is doing matrix multiplication. 

In the scenario of being compute-bound, which happens at higher batch sizes, you will not gain a speed-up using a W4A16 quantized model because the overhead of dequantization will slow down the overall generation. This happens because AWQ quantized models only store the weights in INT4 but perform FP16 operations during inference, so we are essentially converting INT4 -> FP16 during inference.

### Fused modules

Fused modules are a large part of the speedup you get from AutoAWQ. The idea is to combine multiple layers into a single operation, thus becoming more efficient. Fused modules represent a set of custom modules that work separately from Huggingface models. They are compatible with `model.generate()` and other Huggingface methods, which comes with some inflexibility in how you can use your model if you activate fused modules:

- Fused modules are activated when you use `fuse_layers=True`.
- A custom cache is implemented. It preallocates based on batch size and sequence length.
Casper's avatar
Casper committed
106
    - You cannot change the sequence length after you have created your model.
Casper's avatar
Casper committed
107
108
109
    - Reference: `AutoAWQForCausalLM.from_quantized(max_new_tokens=seq_len, batch_size=batch_size)`
- The main accelerator in the fused modules comes from FasterTransformer, which is only compatible with Linux.
- The `past_key_values` from `model.generate()` are only dummy values, so they cannot be used after generation.
110
111
112

### Examples

Casper's avatar
Casper committed
113
114
More examples can be found in the [examples directory](examples).

Casper's avatar
Casper committed
115
<details>
Casper Hansen's avatar
Casper Hansen committed
116

Casper's avatar
Casper committed
117
<summary>Quantization</summary>
Casper Hansen's avatar
Casper Hansen committed
118

119
120
Expect this to take 10-15 minutes on smaller 7B models, and around 1 hour for 70B models.

Casper's avatar
Casper committed
121
```python
Casper's avatar
Casper committed
122
from awq import AutoAWQForCausalLM
Casper's avatar
Casper committed
123
from transformers import AutoTokenizer
Casper Hansen's avatar
Casper Hansen committed
124

Casper's avatar
Casper committed
125
126
model_path = 'lmsys/vicuna-7b-v1.5'
quant_path = 'vicuna-7b-v1.5-awq'
Casper's avatar
Casper committed
127
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
Ji Lin's avatar
Ji Lin committed
128

Casper's avatar
Casper committed
129
130
131
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
Ji Lin's avatar
Ji Lin committed
132

Casper's avatar
Casper committed
133
134
135
136
137
138
# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
Ji Lin's avatar
Ji Lin committed
139
140
```

Casper's avatar
Casper committed
141
142
143
</details>

<details>
Ji Lin's avatar
Ji Lin committed
144

Casper's avatar
Casper committed
145
<summary>Inference</summary>
Ji Lin's avatar
Ji Lin committed
146

Casper's avatar
Casper committed
147
```python
Casper's avatar
Casper committed
148
from awq import AutoAWQForCausalLM
Casper's avatar
Casper committed
149
from transformers import AutoTokenizer, TextStreamer
Ji Lin's avatar
Ji Lin committed
150

Casper's avatar
Casper committed
151
quant_path = "TheBloke/zephyr-7B-beta-AWQ"
Ji Lin's avatar
Ji Lin committed
152

Casper's avatar
Casper committed
153
# Load model
Casper's avatar
Casper committed
154
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
Casper's avatar
Casper committed
155
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
Casper's avatar
Casper committed
156
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
Casper's avatar
Casper committed
157
158
159

# Convert prompt to tokens
prompt_template = """\
Casper's avatar
Casper committed
160
161
162
163
164
<|system|>
</s>
<|user|>
{prompt}</s>
<|assistant|>"""
Casper's avatar
Casper committed
165

Casper's avatar
Casper committed
166
167
168
prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"
Casper's avatar
Casper committed
169
170

tokens = tokenizer(
Casper's avatar
Casper committed
171
    prompt_template.format(prompt=prompt), 
Casper's avatar
Casper committed
172
173
174
175
176
177
178
179
180
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=512
)
Casper's avatar
Casper committed
181
```
Ji Lin's avatar
Ji Lin committed
182

Casper's avatar
Casper committed
183
</details>
Ji Lin's avatar
Ji Lin committed
184

Casper's avatar
Casper committed
185
## Benchmarks
Ji Lin's avatar
Ji Lin committed
186

Casper's avatar
Casper committed
187
188
189
190
These benchmarks showcase the speed and memory usage of processing context (prefill) and generating tokens (decoding). The results include speed at various batch sizes and different versions of AWQ kernels. We have aimed to test models fairly using the same benchmarking tool that you can use to reproduce the results. Do note that speed may vary not only between GPUs but also between CPUs. What matters most is a GPU with high memory bandwidth and a CPU with high single core clock speed.

- Tested with AutoAWQ version 0.1.6
- GPU: RTX 4090 (AMD Ryzen 9 7950X)
Casper's avatar
Casper committed
191
- Command: `python examples/benchmark.py --model_path <hf_model> --batch_size 1`
Casper's avatar
Casper committed
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
- 🟢 for GEMV, 🔵 for GEMM, 🔴 for avoid using

| Model Name |  Size    | Version          | Batch Size | Prefill Length | Decode Length | Prefill tokens/s | Decode tokens/s | Memory (VRAM)    |
|------------|----------|------------------|------------|----------------|---------------|------------------|-----------------|------------------|
| Vicuna     |   7B     | 🟢GEMV           | 1          | 64             | 64            | 639.65           | 198.848         | 4.50 GB (19.05%) |
| Vicuna     |   7B     | 🟢GEMV           | 1          | 2048           | 2048          | 1123.63          | 133.191         | 6.15 GB (26.02%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Mistral    |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1093.35          | 156.317         | 4.35 GB (18.41%) |
| Mistral    |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 3897.02          | 114.355         | 5.55 GB (23.48%) |
| Mistral    |   7B     | 🔵GEMM           | 8          | 64             | 64            | 4199.18          | 1185.25         | 4.35 GB (18.41%) |
| Mistral    |   7B     | 🔵GEMM           | 8          | 2048           | 2048          | 3661.46          | 829.754         | 16.82 GB (71.12%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Mistral    |   7B     | 🟢GEMV           | 1          | 64             | 64            | 531.99           | 188.29          | 4.28 GB (18.08%) |
| Mistral    |   7B     | 🟢GEMV           | 1          | 2048           | 2048          | 903.83           | 130.66          | 5.55 GB (23.48%) |
| Mistral    |   7B     | 🔴GEMV           | 8          | 64             | 64            | 897.87           | 486.46          | 4.33 GB (18.31%) |
| Mistral    |   7B     | 🔴GEMV           | 8          | 2048           | 2048          | 884.22           | 411.893         | 16.82 GB (71.12%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| TinyLlama  |   1B     | 🟢GEMV           | 1          | 64             | 64            | 1088.63          | 548.993         | 0.86 GB (3.62%)  |
| TinyLlama  |   1B     | 🟢GEMV           | 1          | 2048           | 2048          | 5178.98          | 431.468         | 2.10 GB (8.89%)  |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Llama 2    |   13B    | 🔵GEMM           | 1          | 64             | 64            | 820.34           | 96.74           | 8.47 GB (35.83%) |
| Llama 2    |   13B    | 🔵GEMM           | 1          | 2048           | 2048          | 2279.41          | 73.8213         | 10.28 GB (43.46%)|
| Llama 2    |   13B    | 🔵GEMM           | 3          | 64             | 64            | 1593.88          | 286.249         | 8.57 GB (36.24%) |
| Llama 2    |   13B    | 🔵GEMM           | 3          | 2048           | 2048          | 2226.7           | 189.573         | 16.90 GB (71.47%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| MPT        |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1079.06          | 161.344         | 3.67 GB (15.51%) |
| MPT        |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 4069.78          | 114.982         | 5.87 GB (24.82%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Falcon     |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1139.93          | 133.585         | 4.47 GB (18.92%) |
| Falcon     |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 2850.97          | 115.73          | 6.83 GB (28.88%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| CodeLlama  |   34B    | 🔵GEMM           | 1          | 64             | 64            | 681.74           | 41.01           | 19.05 GB (80.57%)|
| CodeLlama  |   34B    | 🔵GEMM           | 1          | 2048           | 2048          | 1072.36          | 35.8316         | 20.26 GB (85.68%)|
| ...        |  ...     | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| DeepSeek   |   33B    | 🔵GEMM           | 1          | 64             | 64            | 1160.18          | 40.29           | 18.92 GB (80.00%)|
| DeepSeek   |   33B    | 🔵GEMM           | 1          | 2048           | 2048          | 1012.1           | 34.0093         | 19.87 GB (84.02%)|
228

Ji Lin's avatar
Ji Lin committed
229
230
## Reference

Casper's avatar
Casper committed
231
If you find AWQ useful or relevant to your research, you can cite their [paper](https://arxiv.org/abs/2306.00978):
Ji Lin's avatar
Ji Lin committed
232
233
234
235
236
237
238
239
240

```
@article{lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song},
  journal={arXiv},
  year={2023}
}
```