Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
Megatron-LM
Commits
fc2c81d3
Commit
fc2c81d3
authored
Mar 31, 2023
by
Jared Casper
Browse files
fix bug in uniform activation recompute
parent
035cae2e
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
1 deletion
+2
-1
megatron/model/transformer.py
megatron/model/transformer.py
+2
-1
No files found.
megatron/model/transformer.py
View file @
fc2c81d3
...
@@ -1036,9 +1036,10 @@ class ParallelTransformer(MegatronModule):
...
@@ -1036,9 +1036,10 @@ class ParallelTransformer(MegatronModule):
"""Forward method with activation checkpointing."""
"""Forward method with activation checkpointing."""
def
custom
(
start
,
end
,
is_transformer_engine
=
False
):
def
custom
(
start
,
end
,
is_transformer_engine
=
False
):
def
custom_forward
(
*
args
,
**
kwargs
):
def
custom_forward
(
*
args
,
**
kwargs
):
x_
,
*
args
=
args
for
index
in
range
(
start
,
end
):
for
index
in
range
(
start
,
end
):
layer
=
self
.
_get_layer
(
index
)
layer
=
self
.
_get_layer
(
index
)
x_
=
layer
(
*
args
,
**
kwargs
)
x_
=
layer
(
x_
,
*
args
,
**
kwargs
)
return
x_
return
x_
def
custom_forward_transformer_engine
(
*
args
,
**
kwargs
):
def
custom_forward_transformer_engine
(
*
args
,
**
kwargs
):
return
custom_forward
(
*
args
,
is_first_microbatch
=
is_first_microbatch
,
**
kwargs
)
return
custom_forward
(
*
args
,
is_first_microbatch
=
is_first_microbatch
,
**
kwargs
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment