"docs/vscode:/vscode.git/clone" did not exist on "cd0f0b8873ca66258c0104010be27f3a73ec9169"
Unverified Commit 33d7e89c authored by Lime-Cakes's avatar Lime-Cakes Committed by GitHub
Browse files

Edited attention.py for older xformers (#1270)

Older versions of xformers require query, key, value to be contiguous, this calls .contiguous() on q/k/v before passing to xformers.
parent b3c5e086
......@@ -557,6 +557,9 @@ class CrossAttention(nn.Module):
return hidden_states
def _memory_efficient_attention_xformers(self, query, key, value):
query = query.contiguous()
key = key.contiguous()
value = value.contiguous()
hidden_states = xformers.ops.memory_efficient_attention(query, key, value, attn_bias=None)
hidden_states = self.reshape_batch_dim_to_heads(hidden_states)
return hidden_states
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment