Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
fairscale
Commits
5220f89b
Unverified
Commit
5220f89b
authored
Oct 09, 2020
by
Benjamin Lefaudeux
Committed by
GitHub
Oct 09, 2020
Browse files
[minor] OSS doc fix - add the DDP wrap (#131)
* wrapping the model in DDP in the tutorial * typo
parent
bfd88cad
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
9 additions
and
4 deletions
+9
-4
docs/source/tutorials/oss.rst
docs/source/tutorials/oss.rst
+9
-4
No files found.
docs/source/tutorials/oss.rst
View file @
5220f89b
Optimizer
state
sharding
========================
Using
torch
.
nn
.
parallel
.
DistributedDataParallel
leads
to
some
wasted
communications
,
but
it
is
possible
and
makes
OSS
a
drop
in
solution
in
your
existing
torch
distributed
code
.
Using
torch
.
nn
.
parallel
.
DistributedDataParallel
leads
to
some
wasted
communications
in
the
case
of
OSS
,
but
it
is
possible
and
makes
OSS
a
drop
in
solution
in
your
existing
torch
distributed
code
.
Let
's suppose that your trainer looks like
.. code-block:: python
import torch
from torch.nn.parallel import DistributedDataParallel as DDP
def train(
rank: int,
...
...
@@ -19,6 +21,7 @@ Let's suppose that your trainer looks like
# Problem statement
model = myAwesomeModel().to(rank)
model = DDP(model, device_ids=[rank])
dataloader = mySuperFastDataloader()
loss_ln = myVeryRelevantLoss()
...
...
@@ -50,6 +53,7 @@ Then sharding the optimizer state is merely a matter of wrapping your optimizer
import torch
from fairscale.optim.oss import OSS
from torch.nn.parallel import DistributedDataParallel as DDP
def train(
rank: int,
...
...
@@ -61,6 +65,7 @@ Then sharding the optimizer state is merely a matter of wrapping your optimizer
# Problem statement
model = myAwesomeModel().to(rank)
model = DDP(model, device_ids=[rank])
dataloader = mySuperFastDataloader()
loss_ln = myVeryRelevantLoss()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment