GitLab

Moving gradient division back to after the allreduce.  Empirically, it appears underflow is more of a danger than overflow.

This is consistent with upstream, and safer against overflow.

* beautiful

* IT'S WORKING

* Hopefully fix race condition for fallback hook

* Updating test

* shared_param -> delayed_allreduce

* Adding a safety check

* One more check

* syntax...

* Fix appears to work in Tomasz's example.

* Somehow shared_param got de-enabled again?

* Remove some fp16 examples that don't converge

Default static loss scale of 1.0 (default value) for resnet50 doesn't converge. Either remove example or put static loss scale 128 on it, which is known to converge well.

* Update README.md

* WIP: update to support new RNN backend code

* small refactor

* add test for rnn w/ packed sequences

amp unit tests

* Add reducer class in parallel/distributed.

* Separate DDP and Reducer examples.

* Don't confuse DDP and reducer in example.