Skip to content

GitLab

Unverified Commit 829b9a62 authored by Lucas Wilkinson's avatar Lucas Wilkinson Committed by GitHub
Browse files

[Perf] Dont create unnecessary pooling params (#22876) 


Signed-off-by: default avatarLucas Wilkinson <lwilkins@redhat.com>
parent 540d54ca
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 3 deletions
vllm/v1/worker/gpu_model_runner.py
View file @ 829b9a62
...@@ -341,13 +341,13 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin): ...@@ -341,13 +341,13 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
model_kwargs = dict[str, Any]() model_kwargs = dict[str, Any]()
num_reqs = self.input_batch.num_reqs num_reqs = self.input_batch.num_reqs
pooling_params = self.input_batch.pooling_metadata.pooling_params num_pooling_reqs = len(self.input_batch.pooling_params)
num_pooling_reqs = len(pooling_params)
if num_pooling_reqs == 0: if num_pooling_reqs == 0:
return model_kwargs return model_kwargs
pooling_params = self.input_batch.pooling_metadata.pooling_params
assert num_pooling_reqs == num_reqs assert num_pooling_reqs == num_reqs
token_type_id_requests = dict[int, Any]() token_type_id_requests = dict[int, Any]()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment