Skip to content

GitLab

Unverified Commit 2094062b authored by youkaichao's avatar youkaichao Committed by GitHub
Browse files

[4.5/N] bugfix for quant config in speculative decode (#10007) 


Signed-off-by: default avataryoukaichao <youkaichao@gmail.com>
parent d93478b3
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 0 deletions
vllm/spec_decode/spec_decode_worker.py
View file @ 2094062b
...@@ -61,6 +61,10 @@ def create_spec_worker(*args, **kwargs) -> "SpecDecodeWorker": ...@@ -61,6 +61,10 @@ def create_spec_worker(*args, **kwargs) -> "SpecDecodeWorker":
draft_worker_config = copy.deepcopy(vllm_config) draft_worker_config = copy.deepcopy(vllm_config)
draft_worker_config.model_config = speculative_config.draft_model_config draft_worker_config.model_config = speculative_config.draft_model_config
draft_worker_config.quant_config = VllmConfig._get_quantization_config(
draft_worker_config.model_config,
vllm_config.load_config,
)
draft_worker_config.parallel_config = speculative_config.draft_parallel_config # noqa draft_worker_config.parallel_config = speculative_config.draft_parallel_config # noqa
# TODO allow draft-model specific load config. # TODO allow draft-model specific load config.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment