README.md 13.4 KB
Newer Older
Casper's avatar
Casper committed
1
# AutoAWQ
Ji Lin's avatar
Ji Lin committed
2

Casper's avatar
Casper committed
3
4
<p align="center">
| <a href="https://github.com/casper-hansen/AutoAWQ/issues/32"><b>Roadmap</b></a> | <a href="https://github.com/casper-hansen/AutoAWQ/tree/main/examples"><b>Examples</b></a> | <a href="https://github.com/casper-hansen/AutoAWQ/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22"><b>Issues: Help Wanted</b></a> |
Casper's avatar
Casper committed
5
6
7
8

</p>
<p align="center">
    <a href="https://huggingface.co/models?search=awq">
Casper's avatar
Casper committed
9
        <img alt="Huggingface - Models" src="https://img.shields.io/badge/🤗_1000+_models_available-8A2BE2">
Casper's avatar
Casper committed
10
11
12
13
14
    </a>
    <a href="https://github.com/casper-hansen/AutoAWQ/releases">
        <img alt="GitHub - Releases" src="https://img.shields.io/github/release/casper-hansen/AutoAWQ.svg">
    </a>
    <a href="https://pypi.org/project/autoawq/">
Casper's avatar
Casper committed
15
        <img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/autoawq/month">
Casper's avatar
Casper committed
16
    </a>
Casper's avatar
Casper committed
17
</p>
Ji Lin's avatar
Ji Lin committed
18

Casper's avatar
Casper committed
19
AutoAWQ is an easy-to-use package for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs.  AutoAWQ was created and improved upon from the [original work](https://github.com/mit-han-lab/llm-awq) from MIT.
Ji Lin's avatar
Ji Lin committed
20

Casper's avatar
Casper committed
21
*Latest News* 🔥
Casper's avatar
Casper committed
22
- [2023/11] AutoAWQ inference has been integrated into 🤗 transformers. Now includes CUDA 12.1 wheels.
Casper's avatar
Casper committed
23
- [2023/10] Mistral (Fused Modules), Bigcode, Turing support, Memory Bug Fix (Saves 2GB VRAM)
Casper Hansen's avatar
Casper Hansen committed
24
- [2023/09] 1.6x-2.5x speed boost on fused models (now including MPT and Falcon).
Casper's avatar
Casper committed
25
26
- [2023/09] Multi-GPU support, bug fixes, and better benchmark scripts available
- [2023/08] PyPi package released and AutoModel class available
Ji Lin's avatar
Ji Lin committed
27
28
29

## Install

Casper's avatar
Casper committed
30
### Prerequisites
Casper's avatar
Casper committed
31

Casper's avatar
Casper committed
32
33
- Your GPU(s) must be of Compute Capability 7.5. Turing and later architectures are supported.
- Your CUDA version must be CUDA 11.8 or later.
Casper's avatar
Casper committed
34

Casper's avatar
Casper committed
35
### Install from PyPi
Casper's avatar
Casper committed
36

Casper's avatar
Casper committed
37
To install the newest AutoAWQ from PyPi, you need CUDA 12.1 installed.
Casper's avatar
Casper committed
38
39

```
Casper's avatar
Casper committed
40
pip install autoawq
Casper's avatar
Casper committed
41
42
```

Casper's avatar
Casper committed
43
If you cannot use CUDA 12.1, you can still use CUDA 11.8 and install the wheel from the [latest release](https://github.com/casper-hansen/AutoAWQ/releases).
Casper's avatar
Casper committed
44
45
46
47
48

```
pip install https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl
```

Casper's avatar
Casper committed
49
### Build from source
Casper's avatar
Casper committed
50

Casper's avatar
Casper committed
51
Build time can take 10-20 minutes. Download your model while you install AutoAWQ.
Casper's avatar
Casper committed
52

Ji Lin's avatar
Ji Lin committed
53
```
Casper's avatar
Casper committed
54
git clone https://github.com/casper-hansen/AutoAWQ
Casper's avatar
Casper committed
55
cd AutoAWQ
Ji Lin's avatar
Ji Lin committed
56
57
58
pip install -e .
```

Casper's avatar
Casper committed
59
## Supported models
Casper Hansen's avatar
Casper Hansen committed
60

Casper's avatar
Casper committed
61
The detailed support list:
Haotian (Ken) Tang's avatar
Haotian (Ken) Tang committed
62

Casper's avatar
Casper committed
63
64
65
66
| Models   | Sizes                       |
| ---------| ----------------------------|
| LLaMA-2  | 7B/13B/70B                  |
| LLaMA    | 7B/13B/30B/65B              |
Casper's avatar
Casper committed
67
| Mistral  | 7B                          |
Casper's avatar
Casper committed
68
69
70
71
72
| Vicuna   | 7B/13B                      |
| MPT      | 7B/30B                      |
| Falcon   | 7B/40B                      |
| OPT      | 125m/1.3B/2.7B/6.7B/13B/30B |
| Bloom    | 560m/3B/7B/                 |
Casper's avatar
Casper committed
73
| GPTJ     | 6.7B                        |
ldwang's avatar
ldwang committed
74
75
| Aquila   | 7B                          |
| Aquila2  | 7B/34B                      |
Casper's avatar
Casper committed
76
77
78
79
80
| Yi       | 6B/34B                      |
| Qwen     | 1.8B/7B/14B/72B             |
| BigCode  | 1B/7B/15B                   |
| GPT NeoX | 20B                         |
| GPT-J    | 6B                          |
81
| Llava    | 7B/13B                      |
Casper's avatar
Casper committed
82
83

## Usage
Ji Lin's avatar
Ji Lin committed
84

Casper's avatar
Casper committed
85
86
Under examples, you can find examples of how to quantize, run inference, and benchmark AutoAWQ models.

87
88
### INT4 GEMM vs INT4 GEMV vs FP16

Casper's avatar
Casper committed
89
There are two versions of AWQ: GEMM and GEMV. Both names relate to how matrix multiplication runs under the hood. We suggest the following:
90

Casper's avatar
Casper committed
91
- GEMV (quantized): 20% faster than GEMM, only batch size 1 (not good for large context).
Casper's avatar
Casper committed
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
- GEMM (quantized): Much faster than FP16 at batch sizes below 8 (good with large contexts).
- FP16 (non-quantized): Recommended for highest throughput: [vLLM](https://github.com/vllm-project/vllm).

#### Compute-bound vs Memory-bound

At small batch sizes with small 7B models, we are memory-bound. This means we are bound by the bandwidth our GPU has to push around the weights in memory, and this is essentially what limits how many tokens per second we can generate. Being memory-bound is what makes quantized models faster because your weights are 3x smaller and can therefore be pushed around in memory much faster. This is different from being compute-bound where the main time spent during generation is doing matrix multiplication. 

In the scenario of being compute-bound, which happens at higher batch sizes, you will not gain a speed-up using a W4A16 quantized model because the overhead of dequantization will slow down the overall generation. This happens because AWQ quantized models only store the weights in INT4 but perform FP16 operations during inference, so we are essentially converting INT4 -> FP16 during inference.

### Fused modules

Fused modules are a large part of the speedup you get from AutoAWQ. The idea is to combine multiple layers into a single operation, thus becoming more efficient. Fused modules represent a set of custom modules that work separately from Huggingface models. They are compatible with `model.generate()` and other Huggingface methods, which comes with some inflexibility in how you can use your model if you activate fused modules:

- Fused modules are activated when you use `fuse_layers=True`.
- A custom cache is implemented. It preallocates based on batch size and sequence length.
Casper's avatar
Casper committed
107
    - You cannot change the sequence length after you have created your model.
Casper's avatar
Casper committed
108
109
110
    - Reference: `AutoAWQForCausalLM.from_quantized(max_new_tokens=seq_len, batch_size=batch_size)`
- The main accelerator in the fused modules comes from FasterTransformer, which is only compatible with Linux.
- The `past_key_values` from `model.generate()` are only dummy values, so they cannot be used after generation.
111
112
113

### Examples

Casper's avatar
Casper committed
114
115
More examples can be found in the [examples directory](examples).

Casper's avatar
Casper committed
116
<details>
Casper Hansen's avatar
Casper Hansen committed
117

Casper's avatar
Casper committed
118
<summary>Quantization</summary>
Casper Hansen's avatar
Casper Hansen committed
119

120
121
Expect this to take 10-15 minutes on smaller 7B models, and around 1 hour for 70B models.

Casper's avatar
Casper committed
122
```python
Casper's avatar
Casper committed
123
from awq import AutoAWQForCausalLM
Casper's avatar
Casper committed
124
from transformers import AutoTokenizer
Casper Hansen's avatar
Casper Hansen committed
125

Casper's avatar
Casper committed
126
127
model_path = 'lmsys/vicuna-7b-v1.5'
quant_path = 'vicuna-7b-v1.5-awq'
Casper's avatar
Casper committed
128
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
Ji Lin's avatar
Ji Lin committed
129

Casper's avatar
Casper committed
130
131
132
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
Ji Lin's avatar
Ji Lin committed
133

Casper's avatar
Casper committed
134
135
136
137
138
139
# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
Ji Lin's avatar
Ji Lin committed
140
141
```

Casper's avatar
Casper committed
142
143
144
</details>

<details>
Ji Lin's avatar
Ji Lin committed
145

Casper's avatar
Casper committed
146
<summary>Inference</summary>
Ji Lin's avatar
Ji Lin committed
147

Casper's avatar
Casper committed
148
```python
Casper's avatar
Casper committed
149
from awq import AutoAWQForCausalLM
Casper's avatar
Casper committed
150
from transformers import AutoTokenizer, TextStreamer
Ji Lin's avatar
Ji Lin committed
151

Casper's avatar
Casper committed
152
quant_path = "TheBloke/zephyr-7B-beta-AWQ"
Ji Lin's avatar
Ji Lin committed
153

Casper's avatar
Casper committed
154
# Load model
Casper's avatar
Casper committed
155
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
Casper's avatar
Casper committed
156
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
Casper's avatar
Casper committed
157
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
Casper's avatar
Casper committed
158
159
160

# Convert prompt to tokens
prompt_template = """\
Casper's avatar
Casper committed
161
162
163
164
165
<|system|>
</s>
<|user|>
{prompt}</s>
<|assistant|>"""
Casper's avatar
Casper committed
166

Casper's avatar
Casper committed
167
168
169
prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"
Casper's avatar
Casper committed
170
171

tokens = tokenizer(
Casper's avatar
Casper committed
172
    prompt_template.format(prompt=prompt), 
Casper's avatar
Casper committed
173
174
175
176
177
178
179
180
181
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=512
)
Casper's avatar
Casper committed
182
```
Ji Lin's avatar
Ji Lin committed
183

Casper's avatar
Casper committed
184
</details>
Ji Lin's avatar
Ji Lin committed
185

Casper's avatar
Casper committed
186
## Benchmarks
Ji Lin's avatar
Ji Lin committed
187

Casper's avatar
Casper committed
188
189
190
191
These benchmarks showcase the speed and memory usage of processing context (prefill) and generating tokens (decoding). The results include speed at various batch sizes and different versions of AWQ kernels. We have aimed to test models fairly using the same benchmarking tool that you can use to reproduce the results. Do note that speed may vary not only between GPUs but also between CPUs. What matters most is a GPU with high memory bandwidth and a CPU with high single core clock speed.

- Tested with AutoAWQ version 0.1.6
- GPU: RTX 4090 (AMD Ryzen 9 7950X)
Casper's avatar
Casper committed
192
- Command: `python examples/benchmark.py --model_path <hf_model> --batch_size 1`
Casper's avatar
Casper committed
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
- 🟢 for GEMV, 🔵 for GEMM, 🔴 for avoid using

| Model Name |  Size    | Version          | Batch Size | Prefill Length | Decode Length | Prefill tokens/s | Decode tokens/s | Memory (VRAM)    |
|------------|----------|------------------|------------|----------------|---------------|------------------|-----------------|------------------|
| Vicuna     |   7B     | 🟢GEMV           | 1          | 64             | 64            | 639.65           | 198.848         | 4.50 GB (19.05%) |
| Vicuna     |   7B     | 🟢GEMV           | 1          | 2048           | 2048          | 1123.63          | 133.191         | 6.15 GB (26.02%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Mistral    |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1093.35          | 156.317         | 4.35 GB (18.41%) |
| Mistral    |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 3897.02          | 114.355         | 5.55 GB (23.48%) |
| Mistral    |   7B     | 🔵GEMM           | 8          | 64             | 64            | 4199.18          | 1185.25         | 4.35 GB (18.41%) |
| Mistral    |   7B     | 🔵GEMM           | 8          | 2048           | 2048          | 3661.46          | 829.754         | 16.82 GB (71.12%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Mistral    |   7B     | 🟢GEMV           | 1          | 64             | 64            | 531.99           | 188.29          | 4.28 GB (18.08%) |
| Mistral    |   7B     | 🟢GEMV           | 1          | 2048           | 2048          | 903.83           | 130.66          | 5.55 GB (23.48%) |
| Mistral    |   7B     | 🔴GEMV           | 8          | 64             | 64            | 897.87           | 486.46          | 4.33 GB (18.31%) |
| Mistral    |   7B     | 🔴GEMV           | 8          | 2048           | 2048          | 884.22           | 411.893         | 16.82 GB (71.12%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| TinyLlama  |   1B     | 🟢GEMV           | 1          | 64             | 64            | 1088.63          | 548.993         | 0.86 GB (3.62%)  |
| TinyLlama  |   1B     | 🟢GEMV           | 1          | 2048           | 2048          | 5178.98          | 431.468         | 2.10 GB (8.89%)  |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Llama 2    |   13B    | 🔵GEMM           | 1          | 64             | 64            | 820.34           | 96.74           | 8.47 GB (35.83%) |
| Llama 2    |   13B    | 🔵GEMM           | 1          | 2048           | 2048          | 2279.41          | 73.8213         | 10.28 GB (43.46%)|
| Llama 2    |   13B    | 🔵GEMM           | 3          | 64             | 64            | 1593.88          | 286.249         | 8.57 GB (36.24%) |
| Llama 2    |   13B    | 🔵GEMM           | 3          | 2048           | 2048          | 2226.7           | 189.573         | 16.90 GB (71.47%)|
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| MPT        |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1079.06          | 161.344         | 3.67 GB (15.51%) |
| MPT        |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 4069.78          | 114.982         | 5.87 GB (24.82%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| Falcon     |   7B     | 🔵GEMM           | 1          | 64             | 64            | 1139.93          | 133.585         | 4.47 GB (18.92%) |
| Falcon     |   7B     | 🔵GEMM           | 1          | 2048           | 2048          | 2850.97          | 115.73          | 6.83 GB (28.88%) |
| ...        |   ...    | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| CodeLlama  |   34B    | 🔵GEMM           | 1          | 64             | 64            | 681.74           | 41.01           | 19.05 GB (80.57%)|
| CodeLlama  |   34B    | 🔵GEMM           | 1          | 2048           | 2048          | 1072.36          | 35.8316         | 20.26 GB (85.68%)|
| ...        |  ...     | ...              | ...        | ...            | ...           | ...              | ...             | ...              |
| DeepSeek   |   33B    | 🔵GEMM           | 1          | 64             | 64            | 1160.18          | 40.29           | 18.92 GB (80.00%)|
| DeepSeek   |   33B    | 🔵GEMM           | 1          | 2048           | 2048          | 1012.1           | 34.0093         | 19.87 GB (84.02%)|
229

Ji Lin's avatar
Ji Lin committed
230
231
## Reference

Casper's avatar
Casper committed
232
If you find AWQ useful or relevant to your research, you can cite their [paper](https://arxiv.org/abs/2306.00978):
Ji Lin's avatar
Ji Lin committed
233
234
235
236
237
238
239
240
241

```
@article{lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song},
  journal={arXiv},
  year={2023}
}
```