Skip to content

GitLab

Unverified Commit 13dd93c6 authored by Russell Bryant's avatar Russell Bryant Committed by GitHub
Browse files

[Core] Force PIECEWISE CUDAGraph mode for encoder-decoder (#25701) 


Signed-off-by: default avatarRussell Bryant <rbryant@redhat.com>
parent 53a30845
Show whitespace changes
Inline Side-by-side
Showing with 4 additions and 2 deletions
vllm/config/__init__.py
View file @ 13dd93c6
...@@ -364,9 +364,11 @@ class VllmConfig: ...@@ -364,9 +364,11 @@ class VllmConfig:
self.compilation_config.cudagraph_mode = \ self.compilation_config.cudagraph_mode = \
CUDAGraphMode.FULL_AND_PIECEWISE CUDAGraphMode.FULL_AND_PIECEWISE
# pooling model does not support full cudagraphs # pooling models and encoder-decoder models
# do not support full cudagraphs
if self.model_config is not None and \ if self.model_config is not None and \
self.model_config.pooler_config is not None: (self.model_config.pooler_config is not None
or self.model_config.is_encoder_decoder):
self.compilation_config.cudagraph_mode = \ self.compilation_config.cudagraph_mode = \
CUDAGraphMode.PIECEWISE CUDAGraphMode.PIECEWISE
else: else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment