Skip to content

GitLab

Unverified Commit 4a5299c9 authored by Tomas Ruiz's avatar Tomas Ruiz Committed by GitHub
Browse files

feat: spec decode with draft models (#24322) 


Signed-off-by: default avatarTomas Ruiz <tomas.ruiz.te@gmail.com>
parent 73f2a81c
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
vllm/v1/worker/utils.py
View file @ 4a5299c9
...@@ -352,8 +352,8 @@ def bind_kv_cache( ...@@ -352,8 +352,8 @@ def bind_kv_cache(
pass pass
else: else:
raise NotImplementedError raise NotImplementedError
layer_name = layer_names[0] for layer_name in layer_names:
runner_kv_caches.append(kv_caches[layer_name]) runner_kv_caches.append(kv_caches[layer_name])
# Bind kv_caches to forward context # Bind kv_caches to forward context
for layer_name, kv_cache in kv_caches.items(): for layer_name, kv_cache in kv_caches.items():
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment