Skip to content

GitLab

Unverified Commit 2cb18428 authored by Daniël de Kok's avatar Daniël de Kok Committed by GitHub
Browse files

`server quantize`: expose groupsize option (#2225)

parent 06d0e880
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 1 deletion
server/text_generation_server/cli.py
View file @ 2cb18428
...@@ -341,6 +341,7 @@ def quantize( ...@@ -341,6 +341,7 @@ def quantize(
upload_to_model_id: Optional[str] = None, upload_to_model_id: Optional[str] = None,
percdamp: float = 0.01, percdamp: float = 0.01,
act_order: bool = False, act_order: bool = False,
groupsize: int = 128,
): ):
if revision is None: if revision is None:
revision = "main" revision = "main"
...@@ -355,7 +356,7 @@ def quantize( ...@@ -355,7 +356,7 @@ def quantize(
quantize( quantize(
model_id=model_id, model_id=model_id,
bits=4, bits=4,
groupsize=128, groupsize=groupsize,
output_dir=output_dir, output_dir=output_dir,
revision=revision, revision=revision,
trust_remote_code=trust_remote_code, trust_remote_code=trust_remote_code,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment