GitLab

* move previous fused_adam and fp16_optimizer to contrib

* make build contrib.fused_adam optional

* change build option name

* remove unnecessary try import

* increase atol for Half-Float comparison to 1.5e-4

* disable tests for different opt_levels

* reset atol

* add bitwise accurate comparison

* Pushing for build tests

* Contrib files

* Removing deprecated checks

Remove torch.autograd.Variable

* move import of amp_C to __init__()

* make fp16/32 separate lists to support mixed param types, disable double test

* make zero_grad consistent between adam/novograd/lamb

* add state_dict, load_state_dict

* add test_restoring, test_loss_scale_decrease

* disable amp outputs for checkpoint tests

* add test for amp.state_dict, cleanup

* add state_dict patch, add test

* fixed testing, cleanup

* add readme for checkpointing

* add docs to source/amp

* add review changes to doc

correctly not apply bias correction to epsilon(same as recent upstream change)
correctly not apply bias correction to weight decay(consistent with upstream AdamW)
Make adam_w_mode for FusedAdam/LAMB, to do L2 or Weight Decay (Adam vs AdamW)
Correct document reg_inside_moment differently from adam_w_mode in FusedNovoGrad
Removed legacy eps_mode from FusedAdam
Make internal math type float across fused optimizers

[flake8](http://flake8.pycqa.org) testing of https://github.com/NVIDIA/apex on Python 3.7.1

$ __flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics__
```
./apex/amp/frontend.py:34:21: F821 undefined name 'self'
            if k in self.options:
                    ^
./apex/amp/frontend.py:35:17: F821 undefined name 'self'
                self.options[k] = v
                ^
./tests/L0/run_mixed_adam/test_fp16_optimizer.py:133:19: F821 undefined name 'os'
    script_path = os.path.dirname(os.path.realpath(__file__))
                  ^
./tests/L0/run_mixed_adam/test_fp16_optimizer.py:133:35: F821 undefined name 'os'
    script_path = os.path.dirname(os.path.realpath(__file__))
                                  ^
./docs/source/conf.py:210:5: F821 undefined name 'List'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
./docs/source/conf.py:210:5: F821 undefined name 'unicode'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
./docs/source/conf.py:210:5: F821 undefined name 'Tuple'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
7     F821 undefined name 'self'
7
```
__E901,E999,F821,F822,F823__ are the "_showstopper_" [flake8](http://flake8.pycqa.org) issues that can halt the runtime with a SyntaxError, NameError, etc. These 5 are different from most other flake8 issues which are merely "style violations" -- useful for readability but they do not effect runtime safety.
* F821: undefined name `name`
* F822: undefined name `name` in `__all__`
* F823: local variable name referenced before assignment
* E901: SyntaxError or IndentationError
* E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

FusedSGD now work as before
FusedAdam now work with o1/o2, no longer fuse scaling and casting
Removed special backend handling for FusedAdam
Moved and updated test for FusedAdam into run_optimizers
Removed legacy tests for optimizers.FP16_optimizer and FusedAdam in run_mixed_adam

Co-authored-by: Aditya Agrawal <aditya.iitb@gmail.com>
Co-authored-by: Marek Kolodziej <mkolod@gmail.com>

* initial commit

* add default O1 mode, enable other modes, add README

* add carilli's review suggestions to README

* Bug fix for non-affine layer-norm + add backward unit test

* clean up tests and add tests for a large batch

fix fused layer norm for >65535 batch

Update FusedLayerNorm for new function API