GitLab

* make the basic example usable out of the box
* clarify

* testing median and MAD

* synchronize on kernels to make sure that we're measuring the actual completion time

* adjusting the circleci threshold, not that the speed has regressed but because we measure proper cuda execution time

* rewrite using autograd and Variable execution queue to make the reduce automatic
* share buckets with OSS to remove duplication
* some speed still likely on the table since the speed vs. bucketing does not match expectations, could be a follow up

Fixes #190

* reverting a change which slipped in #188

* Add CPU support for pipe.py benchmarks, CUDA-free

* adding a shard-aware GradScaler wrap, credits to Sean Naren for the idea
* adding stubs & explanations in the documentation

- removed experimental warning as we have validated it on cifar and
imagenet, transformer is looking good so far too.
- fixed API doc formatting
- make it consistent with the other code in the repo
- tested by making the doc locally and inspect the results

add a clip gradients util, equivalent to torch's but aware of the sharded states. Add a corresponding unit test

* now works on a machine without cuda, easier to debug and quick test

Adds support for:
* Reused layers (e.g. for weight sharing)
* Lazily-constructed layers
* Single-process control via PipeRPCWrapper
* PipelineStyle.AsyncScheudle, which lays the foundation for asynchronous pipeline work by introducing an event loop for each rank/worker to process either activations or gradients as they arrive

Also added examples for multi-process and PipeRPCWrapper

* oss benchmark: add an --amp option
* add a circleCI test

* Some ease of use in the benchmark tool, add a debug option

* small refactor, getting rid of the while loop

fix broken link

- Aurick noticed this bug and I ran into it yesterday
- after the fix, our cifar training shows same gain values from
  different replics now:

```
20-Oct-20 16:00:19 - DEBUG - rank1 - scale 2, gain ratio 1.3512124098087777
20-Oct-20 16:00:19 - DEBUG - rank0 - scale 2, gain ratio 1.3512124098087777
20-Oct-20 16:00:19 - DEBUG - rank1 - timing: data 0:00:00.000600 fwd 0:00:00.003678 loss 0:00:00.000086 bwd 0:00:00.314158 update 0:00:00.002132 rest 0:00:00.000399
20-Oct-20 16:00:19 - DEBUG - rank0 - timing: data 0:00:00.000643 fwd 0:00:00.003460 loss 0:00:00.000084 bwd 0:00:00.314678 update 0:00:00.002001 rest 0:00:00.000408
20-Oct-20 16:00:19 - DEBUG - rank1 - scale 2, gain ratio 1.3514997779980324
20-Oct-20 16:00:19 - DEBUG - rank0 - scale 2, gain ratio 1.3514997779980324
20-Oct-20 16:00:19 - DEBUG - rank1 - timing: data 0:00:00.000732 fwd 0:00:00.003689 loss 0:00:00.000086 bwd 0:00:00.314176 update 0:00:00.002146 rest 0:00:00.000397
20-Oct-20 16:00:19 - DEBUG - rank0 - timing: data 0:00:00.000646 fwd 0:00:00.003542 loss 0:00:00.000089 bwd 0:00:00.314549 update 0:00:00.001956 rest 0:00:00.000392
20-Oct-20 16:00:19 - DEBUG - rank1 - scale 2, gain ratio 1.352149646693932
20-Oct-20 16:00:19 - DEBUG - rank0 - scale 2, gain ratio 1.352149646693932
```

* switching to MNIST
* updating the reference values, should be good to go
* download dataset once for all processes

fix max depth

fix maxdepth

* wip_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_oss_example

* working prototype

* added tutorial script

* update tutorial

* Update mnist_test_oss.py

* Update mnist_test_oss.py

* Update mnist_test_oss.py

* Update mnist_test_pipe.py

* Update tutorial_oss.py

* Update tutorial_pipe.py

* Update tutorial_pipe.py

* Update mnist_test_oss.py

* Update tutorial_pipe.py

* Update mnist_test_pipe.py

* Update tutorial_pipe.py

* fix black

* fix flacke8

* general fixes

* add example oss+pipe

* fix isort

* Update mnist_test_pipe.py

* fix black
Co-authored-by: Vittorio Caggiano <caggiano@devfair0253.h2.fair>

* Minor, ease of life to debug and makes it possible to test a host of models with the same code