GitLab

correctly not apply bias correction to epsilon(same as recent upstream change)
correctly not apply bias correction to weight decay(consistent with upstream AdamW)
Make adam_w_mode for FusedAdam/LAMB, to do L2 or Weight Decay (Adam vs AdamW)
Correct document reg_inside_moment differently from adam_w_mode in FusedNovoGrad
Removed legacy eps_mode from FusedAdam
Make internal math type float across fused optimizers

FusedSGD now work as before
FusedAdam now work with o1/o2, no longer fuse scaling and casting
Removed special backend handling for FusedAdam
Moved and updated test for FusedAdam into run_optimizers
Removed legacy tests for optimizers.FP16_optimizer and FusedAdam in run_mixed_adam

Co-authored-by: Aditya Agrawal <aditya.iitb@gmail.com>
Co-authored-by: Marek Kolodziej <mkolod@gmail.com>

* initial commit

* add default O1 mode, enable other modes, add README

* add carilli's review suggestions to README

* Bug fix for non-affine layer-norm + add backward unit test

* clean up tests and add tests for a large batch

fix fused layer norm for >65535 batch

Update FusedLayerNorm for new function API

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Fixing rank mapping for bn_group size == 8

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

Signed-off-by: Edward Z. Yang <ezyang@fb.com>