GitLab

* move previous fused_adam and fp16_optimizer to contrib

* make build contrib.fused_adam optional

* change build option name

* remove unnecessary try import

* Pushing for build tests

* Contrib files

* Removing deprecated checks

Remove torch.autograd.Variable

* move import of amp_C to __init__()

* make fp16/32 separate lists to support mixed param types, disable double test

* make zero_grad consistent between adam/novograd/lamb

* add state_dict, load_state_dict

* add test_restoring, test_loss_scale_decrease

* disable amp outputs for checkpoint tests

* add test for amp.state_dict, cleanup

* add state_dict patch, add test

* fixed testing, cleanup

* add readme for checkpointing

* add docs to source/amp

* add review changes to doc

correctly not apply bias correction to epsilon(same as recent upstream change)
correctly not apply bias correction to weight decay(consistent with upstream AdamW)
Make adam_w_mode for FusedAdam/LAMB, to do L2 or Weight Decay (Adam vs AdamW)
Correct document reg_inside_moment differently from adam_w_mode in FusedNovoGrad
Removed legacy eps_mode from FusedAdam
Make internal math type float across fused optimizers

[flake8](http://flake8.pycqa.org) testing of https://github.com/NVIDIA/apex on Python 3.7.1

$ __flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics__
```
./apex/amp/frontend.py:34:21: F821 undefined name 'self'
            if k in self.options:
                    ^
./apex/amp/frontend.py:35:17: F821 undefined name 'self'
                self.options[k] = v
                ^
./tests/L0/run_mixed_adam/test_fp16_optimizer.py:133:19: F821 undefined name 'os'
    script_path = os.path.dirname(os.path.realpath(__file__))
                  ^
./tests/L0/run_mixed_adam/test_fp16_optimizer.py:133:35: F821 undefined name 'os'
    script_path = os.path.dirname(os.path.realpath(__file__))
                                  ^
./docs/source/conf.py:210:5: F821 undefined name 'List'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
./docs/source/conf.py:210:5: F821 undefined name 'unicode'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
./docs/source/conf.py:210:5: F821 undefined name 'Tuple'
    # type: (List, unicode, Tuple) -> nodes.field
    ^
7     F821 undefined name 'self'
7
```
__E901,E999,F821,F822,F823__ are the "_showstopper_" [flake8](http://flake8.pycqa.org) issues that can halt the runtime with a SyntaxError, NameError, etc. These 5 are different from most other flake8 issues which are merely "style violations" -- useful for readability but they do not effect runtime safety.
* F821: undefined name `name`
* F822: undefined name `name` in `__all__`
* F823: local variable name referenced before assignment
* E901: SyntaxError or IndentationError
* E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

FusedSGD now work as before
FusedAdam now work with o1/o2, no longer fuse scaling and casting
Removed special backend handling for FusedAdam
Moved and updated test for FusedAdam into run_optimizers
Removed legacy tests for optimizers.FP16_optimizer and FusedAdam in run_mixed_adam

Co-authored-by: Aditya Agrawal <aditya.iitb@gmail.com>
Co-authored-by: Marek Kolodziej <mkolod@gmail.com>

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

fixing empty return from python implementation
  adding proper test to verify functional correctness for python implementation

https://github.com/NVIDIA/apex/issues/181#issuecomment-507958143