GitLab

Handle set/get state for DDP, remove stream which cant be pickled.

* Proper implementation of LARC clipping
 * Documentation of LARC class
 * Modification of FP16_Optimizer to absorb optimizer instance that's being wrapped instead of creating new optimizer instance of same class.

Fixes to validation in imagenet example scripts.

Fixes to validation in imagenet example scripts. Precision and loss reporting modified to be consistent with train.

Fix to imagenet main.py data normalization.

* More stringent check for distributed refresh

Fixed deadlock issue

Adding changes due to https://github.com/pytorch/pytorch/pull/8660

Co-authored-by: Michael Carilli <mcarilli@gmail.com>

Type in variable name: hidden --> hiddens (w/ trailing 's')

__hidden__ in an undefined name in this context but __hiddens__ is used on the lines above.

flake8 testing of https://github.com/NVIDIA/apex on Python 3.6.3

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./setup.py:50:23: F821 undefined name 'ctypes'
        cudart_path = ctypes.util.find_library('cudart')
                      ^
./apex/amp/wrap.py:194:37: F821 undefined name 'hidden'
            elif utils.is_fp_tensor(hidden):
                                    ^
./apex/amp/wrap.py:195:41: F821 undefined name 'hidden'
                new_args.append(cast_fn(hidden))
                                        ^
./apex/amp/wrap.py:198:33: F821 undefined name 'hidden'
                new_args.append(hidden)
                                ^
4     F821 undefined name 'hidden'
4
```

import ctypes for line 51 of setup.py