Skip to content

GitLab

Unverified Commit 21e39436 authored by Calvin Chen's avatar Calvin Chen Committed by GitHub
Browse files

[XPU] fix xpu to set cudagraph batch sizes (#23044) 


Signed-off-by: default avatarcalvin chen <wen.chen@dynamia.ai>
parent 6d243efe
Show whitespace changes
Inline Side-by-side
Showing with 4 additions and 2 deletions
vllm/v1/worker/gpu_model_runner.py
View file @ 21e39436
......@@ -232,6 +232,8 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
# The convention is different.
# self.cudagraph_batch_sizes sorts in ascending order.
# The batch sizes in the config are in descending order.
if self.compilation_config.cudagraph_capture_sizes and \
self.compilation_config.cudagraph_mode != CUDAGraphMode.NONE:
self.cudagraph_batch_sizes = list(
reversed(self.compilation_config.cudagraph_capture_sizes))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment