Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
Megatron-LM
Commits
b0b017ff
Commit
b0b017ff
authored
Aug 18, 2021
by
hyunwoongko
Committed by
mshoeybi
Aug 22, 2021
Browse files
Modify comments
parent
c6b77639
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
2 deletions
+2
-2
megatron/model/fused_softmax.py
megatron/model/fused_softmax.py
+2
-2
No files found.
megatron/model/fused_softmax.py
View file @
b0b017ff
...
@@ -138,8 +138,8 @@ class FusedScaleMaskSoftmax(nn.Module):
...
@@ -138,8 +138,8 @@ class FusedScaleMaskSoftmax(nn.Module):
self
.
scaled_masked_softmax_fusion
# user want to fuse
self
.
scaled_masked_softmax_fusion
# user want to fuse
and
self
.
input_in_float16
# input must be fp16
and
self
.
input_in_float16
# input must be fp16
and
mask
is
not
None
# mask tensor must not be None
and
mask
is
not
None
# mask tensor must not be None
and
16
<
sk
<=
2048
# s
q
must be 16 ~ 2048
and
16
<
sk
<=
2048
# s
k
must be 16 ~ 2048
and
sq
%
4
==
0
# s
k
must be divisor of 4
and
sq
%
4
==
0
# s
q
must be divisor of 4
and
attn_batches
%
4
==
0
# np * b must be divisor of 4
and
attn_batches
%
4
==
0
# np * b must be divisor of 4
):
):
if
0
<=
sk
<=
2048
:
if
0
<=
sk
<=
2048
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment