Skip to content

GitLab

Unverified Commit 21e39436 authored by Calvin Chen's avatar Calvin Chen Committed by GitHub
Browse files

[XPU] fix xpu to set cudagraph batch sizes (#23044) 


Signed-off-by: default avatarcalvin chen <wen.chen@dynamia.ai>
parent 6d243efe
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 2 deletions
vllm/v1/worker/gpu_model_runner.py
View file @ 21e39436
...@@ -232,8 +232,10 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin): ...@@ -232,8 +232,10 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
# The convention is different. # The convention is different.
# self.cudagraph_batch_sizes sorts in ascending order. # self.cudagraph_batch_sizes sorts in ascending order.
# The batch sizes in the config are in descending order. # The batch sizes in the config are in descending order.
self.cudagraph_batch_sizes = list( if self.compilation_config.cudagraph_capture_sizes and \
reversed(self.compilation_config.cudagraph_capture_sizes)) self.compilation_config.cudagraph_mode != CUDAGraphMode.NONE:
self.cudagraph_batch_sizes = list(
reversed(self.compilation_config.cudagraph_capture_sizes))
# Cache the device properties. # Cache the device properties.
self._init_device_properties() self._init_device_properties()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment