GitLab

update SyncBatchNorm API

adding a test and example for sync (group) bn with group_size<world_size

Better FP16 support in pytorch fp16 utils.

This commit adds an FP16Model class as a successor to network_to_half.

The benefits of this class are:

- Preservation of single-precision for BatchNorm layers. The models
  generated by network_to_half() convert BatchNorm moment tensors to
  half-precision, then back to single-precision, which hurts the
  accuracy of the moment estimators and occasionally results in NaNs.
- Support for multi-argument nn.Modules (self-explanatory from code).

Removing patching of loss.backward, which appears to cause memory leaks (reference cycles?) in some models

apex.optimizers.FP16_Optimizer: add state_dict() and load_state_dict()

Restore fused kernel

allowing syncBN to run with affine = False