GitLab

test update to resolve
  https://github.com/NVIDIA/apex/issues/134#issue-403525480

Using identical learning rate for both DDP with sync BN and single process BN.
The previous configure leaves the impression that sync BN requires adjusting lr
in the script, which is not true.

Fix + tests for the eval->training caching issue

patching grid reduction to be volta-safe

[sync BN nhwc]

fixing utility function convert_syncbn_model to accept channel_last flag and properly set attribute for nested layers

Added kernel to support sync BN for channel last tensor

[syncBN]

replacing new_group with torch.distributed.group.WORLD, avoids creating new
group in every iteration.

This should resolve the issue in Training gets stuck when using SyncBN #105

Update __init__.py

Update __init__.py

torch.distributed.new_group and torch.distributed.reduce_op are deprecated on pytorch 1.0.0, this fix can avoid some errors for now.

WIP: improve fused adam

This way optimizer will be more general, and warm up should be handled by user

Adding process group in convert_syncbn_model