GitLab

This commit adds an FP16Model class as a successor to network_to_half.

The benefits of this class are:

- Preservation of single-precision for BatchNorm layers. The models
  generated by network_to_half() convert BatchNorm moment tensors to
  half-precision, then back to single-precision, which hurts the
  accuracy of the moment estimators and occasionally results in NaNs.
- Support for multi-argument nn.Modules (self-explanatory from code).

Update default dims in word_language_model to be multiples of 8 to enable Tensor Core use

Updated default sizes to be multiples of 8 to enable Tensor Core use.  Added performance guidelines to README.

add unit tests for optimizers/fp16_optimizer

don't convert to float bn with affine=False

[syncBN]

test update to resolve
  https://github.com/NVIDIA/apex/issues/134#issue-403525480

Using identical learning rate for both DDP with sync BN and single process BN.
The previous configure leaves the impression that sync BN requires adjusting lr
in the script, which is not true.

Fix + tests for the eval->training caching issue

patching grid reduction to be volta-safe

[sync BN nhwc]

fixing utility function convert_syncbn_model to accept channel_last flag and properly set attribute for nested layers

Added kernel to support sync BN for channel last tensor