GitLab

Update example as style arg was removed in https://github.com/facebookresearch/fairscale/pull/345

* lintfixes

* come on black

* Update tutorial_pipe_multiprocess.py

make RANK global like the other tutorials
Co-authored-by: Vittorio Caggiano <caggiano@gmail.com>

* [feat] add CPU support to tutorials in examples

* now works on a machine without cuda
* fixes some minor typos

* [cleanup] factorize tutorials in examples

* collects duplicate code across tutorials in helpers.py

* [fix] getData in tutorials now returns iterable

* adding a shard-aware GradScaler wrap, credits to Sean Naren for the idea
* adding stubs & explanations in the documentation

add a clip gradients util, equivalent to torch's but aware of the sharded states. Add a corresponding unit test

Adds support for:
* Reused layers (e.g. for weight sharing)
* Lazily-constructed layers
* Single-process control via PipeRPCWrapper
* PipelineStyle.AsyncScheudle, which lays the foundation for asynchronous pipeline work by introducing an event loop for each rank/worker to process either activations or gradients as they arrive

Also added examples for multi-process and PipeRPCWrapper

* wip_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_pipe_example

* [wip]mnist_oss_example

* working prototype

* added tutorial script

* update tutorial

* Update mnist_test_oss.py

* Update mnist_test_oss.py

* Update mnist_test_oss.py

* Update mnist_test_pipe.py

* Update tutorial_oss.py

* Update tutorial_pipe.py

* Update tutorial_pipe.py

* Update mnist_test_oss.py

* Update tutorial_pipe.py

* Update mnist_test_pipe.py

* Update tutorial_pipe.py

* fix black

* fix flacke8

* general fixes

* add example oss+pipe

* fix isort

* Update mnist_test_pipe.py

* fix black
Co-authored-by: Vittorio Caggiano <caggiano@devfair0253.h2.fair>