GitLab

fixing utility function convert_syncbn_model to accept channel_last flag and properly set attribute for nested layers

Added kernel to support sync BN for channel last tensor

[syncBN]

replacing new_group with torch.distributed.group.WORLD, avoids creating new
group in every iteration.

This should resolve the issue in Training gets stuck when using SyncBN #105

Update __init__.py

Update __init__.py

torch.distributed.new_group and torch.distributed.reduce_op are deprecated on pytorch 1.0.0, this fix can avoid some errors for now.

WIP: improve fused adam

This way optimizer will be more general, and warm up should be handled by user

Adding process group in convert_syncbn_model

Update examples to PyTorch >=0.4.0

adjusted kernel config for better perf.
removed divergence in welford warp reduction.

supporting user specified process group

adjusted kernel config for better perf.
removed divergence in welford warp reduction.

* Adding some missing fields to adamopt documentation.

* Adding some clarification to documentation.

* Pre-release of fused layer norm apex extension

* Remove half and __half2 specializations

* Code changes from review