Skip to content

GitLab

Unverified Commit 29799dda authored by Li, Jiang's avatar Li, Jiang Committed by GitHub
Browse files

[Bugfix] Add missing VIT backend dispatch on CPU (#24623) 


Signed-off-by: default avatarjiang1.li <jiang1.li@intel.com>
parent f17a6aa4
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 1 deletion
vllm/attention/layer.py
View file @ 29799dda
...@@ -399,7 +399,8 @@ class MultiHeadAttention(nn.Module): ...@@ -399,7 +399,8 @@ class MultiHeadAttention(nn.Module):
key, key,
value, value,
scale=self.scale) scale=self.scale)
elif self.attn_backend == _Backend.TORCH_SDPA: elif (self.attn_backend == _Backend.TORCH_SDPA
or self.attn_backend == _Backend.TORCH_SDPA_VLLM_V1):
query, key, value = (x.transpose(1, 2) query, key, value = (x.transpose(1, 2)
for x in (query, key, value)) for x in (query, key, value))
out = F.scaled_dot_product_attention(query, out = F.scaled_dot_product_attention(query,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment