GitLab

FusedSGD now work as before
FusedAdam now work with o1/o2, no longer fuse scaling and casting
Removed special backend handling for FusedAdam
Moved and updated test for FusedAdam into run_optimizers
Removed legacy tests for optimizers.FP16_optimizer and FusedAdam in run_mixed_adam

Co-authored-by: Aditya Agrawal <aditya.iitb@gmail.com>
Co-authored-by: Marek Kolodziej <mkolod@gmail.com>

* initial commit

* add default O1 mode, enable other modes, add README

* add carilli's review suggestions to README

* Bug fix for non-affine layer-norm + add backward unit test

* clean up tests and add tests for a large batch

fix fused layer norm for >65535 batch

Update FusedLayerNorm for new function API

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Fixing rank mapping for bn_group size == 8

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

https://github.com/NVIDIA/apex/issues/181#issuecomment-507958143

Add support for fp16 update term (new UPD_T typename in template)