GitLab

Better FP16 support in pytorch fp16 utils.

This commit adds an FP16Model class as a successor to network_to_half.

The benefits of this class are:

- Preservation of single-precision for BatchNorm layers. The models
  generated by network_to_half() convert BatchNorm moment tensors to
  half-precision, then back to single-precision, which hurts the
  accuracy of the moment estimators and occasionally results in NaNs.
- Support for multi-argument nn.Modules (self-explanatory from code).

Removing patching of loss.backward, which appears to cause memory leaks (reference cycles?) in some models

apex.optimizers.FP16_Optimizer: add state_dict() and load_state_dict()

Restore fused kernel

allowing syncBN to run with affine = False

Update default dims in word_language_model to be multiples of 8 to enable Tensor Core use

Updated default sizes to be multiples of 8 to enable Tensor Core use.  Added performance guidelines to README.

add unit tests for optimizers/fp16_optimizer

don't convert to float bn with affine=False

[syncBN]