Skip to content

GitLab

Unverified Commit 8b9ce244 authored by mcarilli's avatar mcarilli Committed by GitHub
Browse files

Update two_gpu_unit_test.py

parent 878ba512
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
tests/synced_batchnorm/two_gpu_unit_test.py
View file @ 8b9ce244
...@@ -94,6 +94,7 @@ out_bn = bn(inp_bn) ...@@ -94,6 +94,7 @@ out_bn = bn(inp_bn)
out_bn.backward(grad_bn) out_bn.backward(grad_bn)
# compensating the averaging over processes done by DDP # compensating the averaging over processes done by DDP
# in order to produce mathematically equivalent result # in order to produce mathematically equivalent result
# https://github.com/NVIDIA/apex/issues/134#issuecomment-458307368
for param in bn.parameters(): for param in bn.parameters():
param.grad = param.grad / args.world_size param.grad = param.grad / args.world_size
bn_opt = optim.SGD(bn.parameters(), lr=1.0) bn_opt = optim.SGD(bn.parameters(), lr=1.0)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment