GitLab

* More stringent check for distributed refresh

Fixed deadlock issue

Adding changes due to https://github.com/pytorch/pytorch/pull/8660

Co-authored-by: Michael Carilli <mcarilli@gmail.com>

Type in variable name: hidden --> hiddens (w/ trailing 's')

__hidden__ in an undefined name in this context but __hiddens__ is used on the lines above.

flake8 testing of https://github.com/NVIDIA/apex on Python 3.6.3

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./setup.py:50:23: F821 undefined name 'ctypes'
        cudart_path = ctypes.util.find_library('cudart')
                      ^
./apex/amp/wrap.py:194:37: F821 undefined name 'hidden'
            elif utils.is_fp_tensor(hidden):
                                    ^
./apex/amp/wrap.py:195:41: F821 undefined name 'hidden'
                new_args.append(cast_fn(hidden))
                                        ^
./apex/amp/wrap.py:198:33: F821 undefined name 'hidden'
                new_args.append(hidden)
                                ^
4     F821 undefined name 'hidden'
4
```

import ctypes for line 51 of setup.py

Handle the use of .sum() in fused LSTM/GRU backward

amp shouldn't convert CPU tensors

DDP fix, imagenet speedup