Skip to content

GitLab

Unverified Commit 8793a9f7 authored by Casper's avatar Casper Committed by GitHub
Browse files

Merge pull request #80 from casper-hansen/low_cpu_mem_example 

Add low_cpu_mem_usage=True in example
parents 1c5ccc79 aa6497cd
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
examples/basic_quant.py
View file @ 8793a9f7
...@@ -7,7 +7,7 @@ quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": ...@@ -7,7 +7,7 @@ quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version":
# Load model # Load model
# NOTE: pass safetensors=True to load safetensors # NOTE: pass safetensors=True to load safetensors
model = AutoAWQForCausalLM.from_pretrained(model_path) model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize # Quantize
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment