Unverified Commit 98c12cff authored by Stas Bekman's avatar Stas Bekman Committed by GitHub
Browse files

[Doc] fix the autoAWQ example (#7937)

parent f52a43a8
...@@ -19,19 +19,21 @@ You can quantize your own models by installing AutoAWQ or picking one of the `40 ...@@ -19,19 +19,21 @@ You can quantize your own models by installing AutoAWQ or picking one of the `40
$ pip install autoawq $ pip install autoawq
After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize Vicuna 7B v1.5: After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
.. code-block:: python .. code-block:: python
from awq import AutoAWQForCausalLM from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer from transformers import AutoTokenizer
model_path = 'lmsys/vicuna-7b-v1.5' model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
quant_path = 'vicuna-7b-v1.5-awq' quant_path = 'mistral-instruct-v0.2-awq'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
# Load model # Load model
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True}) model = AutoAWQForCausalLM.from_pretrained(
model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize # Quantize
...@@ -41,6 +43,8 @@ After installing AutoAWQ, you are ready to quantize a model. Here is an example ...@@ -41,6 +43,8 @@ After installing AutoAWQ, you are ready to quantize a model. Here is an example
model.save_quantized(quant_path) model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path) tokenizer.save_pretrained(quant_path)
print(f'Model is quantized and saved at "{quant_path}"')
To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command: To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command:
.. code-block:: console .. code-block:: console
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment