correct the dimension comment of matrix multiplication (#11494)

Co-authored-by: Frederik Bode <frederik@paperbox.ai>

correct the dimension comment of matrix multiplication (#11494)
Co-authored-by: Frederik Bode <frederik@paperbox.ai>
b43e3f93 · Frederik Bode · GitHub · f37f2adb · b43e3f93
Unverified Commit b43e3f93 authored Apr 30, 2021 by Frederik Bode Committed by GitHub Apr 30, 2021
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

src/transformers/models/longformer/modeling_longformer.py src/transformers/models/longformer/modeling_longformer.py +1 -1

No files found.
--- a/src/transformers/models/longformer/modeling_longformer.py
+++ b/src/transformers/models/longformer/modeling_longformer.py
@@ -814,7 +814,7 @@ class LongformerSelfAttention(nn.Module):
        # matrix multiplication
        # bcxd: batch_size * num_heads x chunks x 2window_overlap x head_dim
        # bcyd: batch_size * num_heads x chunks x 2window_overlap x head_dim
-        # bcxy: batch_size * num_heads x chunks x 2window_overlap x window_overlap
+        # bcxy: batch_size * num_heads x chunks x 2window_overlap x 2window_overlap
        diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
        # convert diagonals into columns