GitLab

* [fix] fixing circleCI for AdaScale

- ran black, isort, flake8, mypy

* more fix

* Add implementation of AdaScale

* add adascale docs

The expert annotation is used by clip_grads and DDP.

* fixing the issue wrt Apex, validated with Latte, Classy would need another pass

* bugfix
* adjust default non-regression loss, not all_reduced now

* wrapping the model in DDP in the tutorial

* typo

More realistic benchmarks, comparing apples to apples. DDP/OSS+DDP/OSS+SDP

* new unit test to catch rank issues in OSS

Currently only implemented for a single process and expert.

authored-by: Naman Goyal <namangoyal@learnfair0755.h2.fair>

* Add unittest for checkpoint & DDP

- this change adds test cases to reproduce the error with checkpoint & DDP
- mandeep mentioned that there is also deadlock in this case, but this
  change doesn't cover that.
- we cover cases where weight sharing is OK
- however, same module multiple checkpoint or find_unused_parameters are
  both not OK

* added norm checks

Same bucketing strategy for OSS and SDP:
sort everything ahead of time, per rank and per size, smaller tensors first. Bucket the smallest elements in a fixed buffer, send async, then send all the others async, and get back to the bucket. Once done then scatter the contents if needed

support optimizer state sharding for megatron

* minor, but gives some memory back
* adjust CI and regression checks to 4 gpu

- adding the buffer broadcast option
- minor cleanup in shardedDDP

add badges and link to readthedoc

- small benchmark refactor, only one for all backends and ddp
- deterministic, enforce alignment with pytorch ddp

* various fixes, no more issues with `make html` and more API fields should be populated

* Broadcasting grad-enabled tensors is forbidden in Gloo, because this is not differentiable. Workaround

* Doc extensions to some APIs
* FIx the benchmark and tutorial

Adds support for distributing pipeline stages across multiple processes (and therefore multiple machines)
* Adds a style argument to the Pipe constructor, defaulting to PipelineStyle.SingleProcess, but also supporting PipelineStyle.MultiProcess
* Added support for lazy construction of modules (see lazy_construction for an example)
* Added two implementations of inter-process communication: one based on rpc with globally visible queues, one based on send/recv
* Copied all the relevant tests from tests/pipe to tests/pipe_process and modified them to exercise PipelineStyle.MultiProcess

- rename oss_ddp to ShardedDataParallel
- some refactoring
- ShardedDataParallel owns the sharded optimizer, exposed if need be
- some small perf bumps

Add a small tutorial, similar to the OSS README

skip python venv files, and coverage reports

Return either the local or global state when queried, depending on a prior consolidation

Make OSS compatible with optimizers which do not support the closure argument