Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
8d17fa63
Unverified
Commit
8d17fa63
authored
Aug 12, 2025
by
Sugar-zsg
Committed by
GitHub
Aug 12, 2025
Browse files
[V0] Correct CUDA Graph capture for encoder-decoder models (#22630)
parent
9f909b89
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
11 additions
and
1 deletion
+11
-1
vllm/config/__init__.py
vllm/config/__init__.py
+11
-1
No files found.
vllm/config/__init__.py
View file @
8d17fa63
...
@@ -1164,8 +1164,18 @@ class ModelConfig:
...
@@ -1164,8 +1164,18 @@ class ModelConfig:
"non-quantized models."
,
self
.
quantization
)
"non-quantized models."
,
self
.
quantization
)
def
_verify_cuda_graph
(
self
)
->
None
:
def
_verify_cuda_graph
(
self
)
->
None
:
# The `max_seq_len_to_capture` was incorrectly
# based on the encoder's input length (448)
# but not the decoder's larger input length (1500).
# This change ensures the CUDA Graph captures the correct,
# larger sequence length, allowing it to work as intended.
effective_max_seq_len
=
self
.
max_model_len
if
self
.
is_encoder_decoder
:
effective_max_seq_len
=
max
(
effective_max_seq_len
,
getattr
(
self
.
hf_config
,
"max_source_positions"
,
0
))
self
.
max_seq_len_to_capture
=
min
(
self
.
max_seq_len_to_capture
,
self
.
max_seq_len_to_capture
=
min
(
self
.
max_seq_len_to_capture
,
self
.
max_model
_len
)
effective_max_seq
_len
)
# CUDAGraph capture not supported for enc-dec models and mllama on ROCm
# CUDAGraph capture not supported for enc-dec models and mllama on ROCm
ROCM_UNSUPPORTED_MODELS
=
[
'mllama'
]
ROCM_UNSUPPORTED_MODELS
=
[
'mllama'
]
unsupported_rocm
=
(
self
.
hf_config
.
model_type
unsupported_rocm
=
(
self
.
hf_config
.
model_type
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment