Skip to content

GitLab

Unverified Commit 0e3bb543 authored by Raushan Turganbay's avatar Raushan Turganbay Committed by GitHub
Browse files

[Bugfix] Support compile for Transformers multimodal (#23095) 


Signed-off-by: default avatarraushan <raushan@huggingface.co>
parent 569aefd1
Hide whitespace changes
Inline Side-by-side
Showing with 7 additions and 0 deletions
vllm/model_executor/models/transformers.py
View file @ 0e3bb543
...@@ -709,6 +709,13 @@ def flatten_and_concat(x: list[torch.Tensor]) -> torch.Tensor: ...@@ -709,6 +709,13 @@ def flatten_and_concat(x: list[torch.Tensor]) -> torch.Tensor:
MultiModalProcessor, MultiModalProcessor,
info=MultiModalProcessingInfo, info=MultiModalProcessingInfo,
dummy_inputs=MultiModalDummyInputsBuilder) dummy_inputs=MultiModalDummyInputsBuilder)
@support_torch_compile(
dynamic_arg_dims={
"input_ids": 0,
"positions": -1,
"intermediate_tensors": 0,
"inputs_embeds": 0,
}) # set `positions` to last dim to support Qwen-mrope
class TransformersForMultimodalLM(TransformersForCausalLM, SupportsMultiModal): class TransformersForMultimodalLM(TransformersForCausalLM, SupportsMultiModal):
# Backwards compatibility for prev released models. State dicts back then # Backwards compatibility for prev released models. State dicts back then
# had different formats and cannot be loaded with `AutoModel` mapping as is # had different formats and cannot be loaded with `AutoModel` mapping as is
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment