Skip to content

GitLab

"docs/source/en/api/parallel.md" did not exist on "0d1d267b12e47b40b0e8f265339c76e0f45f8c49"
Commit 604d423b authored by Thor Johnsen's avatar Thor Johnsen
Browse files

Broadcast parameters from rank 0 just to be safe

parent bc98827b
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
apex/contrib/optimizers/distributed_fused_adam.py
View file @ 604d423b
...@@ -89,6 +89,7 @@ class DistributedFusedAdam(torch.optim.Optimizer): ...@@ -89,6 +89,7 @@ class DistributedFusedAdam(torch.optim.Optimizer):
self._grads_info = [] self._grads_info = []
for group in self.param_groups: for group in self.param_groups:
for p in group['params']: for p in group['params']:
torch.distributed.broadcast(p,0)
if not p.requires_grad: if not p.requires_grad:
continue continue
p_grads_size = p.numel() p_grads_size = p.numel()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment