GitLab

NCCL all_to_all is now supported in PyTorch (since v1.8.0)

Fixes: #548

* tentatively fixing the cpu version of circleci jobs, now pipe tests are the last ones standing
* fixing oss backcompat, trying to fix rpc in old pytorch also
* fixing the file based init in torch 1.5

* file based dist init
* nicer handling of broken world sizes vs. number of available GPUs, do not break but warn out

Adds support for:
* Reused layers (e.g. for weight sharing)
* Lazily-constructed layers
* Single-process control via PipeRPCWrapper
* PipelineStyle.AsyncScheudle, which lays the foundation for asynchronous pipeline work by introducing an event loop for each rank/worker to process either activations or gradients as they arrive

Also added examples for multi-process and PipeRPCWrapper

The expert annotation is used by clip_grads and DDP.

Currently only implemented for a single process and expert.