GitLab

Better FP16 support in pytorch fp16 utils.

This commit adds an FP16Model class as a successor to network_to_half.

The benefits of this class are:

- Preservation of single-precision for BatchNorm layers. The models
  generated by network_to_half() convert BatchNorm moment tensors to
  half-precision, then back to single-precision, which hurts the
  accuracy of the moment estimators and occasionally results in NaNs.
- Support for multi-argument nn.Modules (self-explanatory from code).

Removing patching of loss.backward, which appears to cause memory leaks (reference cycles?) in some models

apex.optimizers.FP16_Optimizer: add state_dict() and load_state_dict()

Restore fused kernel

allowing syncBN to run with affine = False

Update default dims in word_language_model to be multiples of 8 to enable Tensor Core use

Updated default sizes to be multiples of 8 to enable Tensor Core use.  Added performance guidelines to README.

add unit tests for optimizers/fp16_optimizer

don't convert to float bn with affine=False

[syncBN]

test update to resolve
  https://github.com/NVIDIA/apex/issues/134#issue-403525480

Using identical learning rate for both DDP with sync BN and single process BN.
The previous configure leaves the impression that sync BN requires adjusting lr
in the script, which is not true.