Skip to content

GitLab

Unverified Commit b43e3f93 authored by Frederik Bode's avatar Frederik Bode Committed by GitHub
Browse files

correct the dimension comment of matrix multiplication (#11494) 


Co-authored-by: default avatarFrederik Bode <frederik@paperbox.ai>
parent f37f2adb
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
src/transformers/models/longformer/modeling_longformer.py
View file @ b43e3f93
...@@ -814,7 +814,7 @@ class LongformerSelfAttention(nn.Module): ...@@ -814,7 +814,7 @@ class LongformerSelfAttention(nn.Module):
# matrix multiplication # matrix multiplication
# bcxd: batch_size * num_heads x chunks x 2window_overlap x head_dim # bcxd: batch_size * num_heads x chunks x 2window_overlap x head_dim
# bcyd: batch_size * num_heads x chunks x 2window_overlap x head_dim # bcyd: batch_size * num_heads x chunks x 2window_overlap x head_dim
# bcxy: batch_size * num_heads x chunks x 2window_overlap x window_overlap # bcxy: batch_size * num_heads x chunks x 2window_overlap x 2window_overlap
diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply
# convert diagonals into columns # convert diagonals into columns
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment