GitLab

Update default dims in word_language_model to be multiples of 8 to enable Tensor Core use

Updated default sizes to be multiples of 8 to enable Tensor Core use.  Added performance guidelines to README.

add unit tests for optimizers/fp16_optimizer

don't convert to float bn with affine=False

[syncBN]

test update to resolve
  https://github.com/NVIDIA/apex/issues/134#issue-403525480

Using identical learning rate for both DDP with sync BN and single process BN.
The previous configure leaves the impression that sync BN requires adjusting lr
in the script, which is not true.

Fix + tests for the eval->training caching issue

patching grid reduction to be volta-safe

[sync BN nhwc]

fixing utility function convert_syncbn_model to accept channel_last flag and properly set attribute for nested layers

Added kernel to support sync BN for channel last tensor

[syncBN]

replacing new_group with torch.distributed.group.WORLD, avoids creating new
group in every iteration.

This should resolve the issue in Training gets stuck when using SyncBN #105

Update __init__.py

Update __init__.py

torch.distributed.new_group and torch.distributed.reduce_op are deprecated on pytorch 1.0.0, this fix can avoid some errors for now.