GitLab

* Add fairscale.utils.containers
Co-authored-by: Min Xu <24926999+min-xu-ai@users.noreply.github.com>

* Add fairscale.nn.misc.checkpoint_activations
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
Co-authored-by: Min Xu <24926999+min-xu-ai@users.noreply.github.com>
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>

* flat params all along, way simpler
* updating the docstring

fix a broken earlier commit, only worked for the first step

* minor

* minor

cache this iterator, easy speed up

* Adding a proper ddp parity / AMP unit test, overdue
* catch non-AMP pytorch

It is not currently being used so we can simplify the interface
by removing it.

* restoring the regression test, adding a test of the for_each optims
* fix the regression test on circleci
* removing unused flags

* precise skip, only if agent has only cpu

* mp cleanup

* round of multiprocess refactoring

* test golden run

* print cuda stats

* fix lint errors

* enable multiprocess pipe benchmarks

* set world size to be available gpus

* more changes

* use synthetic loaders for intermediate pipeline stages

* merged master

* fix for the devices property

* dataloader fix

* modify rank check

* print wps stats

* enable verification

* fix logging

* fix flag name

* fix flag name

* check for rank

* fix indent

* pass args

* pass args

* modify golden data

* remove unused print messsage

* fix lint errors

* add comments

* fix benchmarks
Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>

* [feat] Add AdaScaleWrapper

- This enables a different API for wrapping an optimizer with AdaScale.
- This also enables AdaScale to be wrapped by OSS.
- However, OSS wrapping AdaScale results in different optimization,
  which future research will be needed to study its effects.

testing: add unit tests.

* addressed comment: typo

* adding the .to(device) support + unit testing
* doc update

* adding a test to prove the inter operability with upstream pytorch
* updating the changelog
* eager state pruning
* pytorch 1.5 compat

* no idea about the root issue, but it proved to be fairly narrowed (gloo+cpu+python3.8+no cuda installed) so I guess that's out of scope for fairscale

authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>

* adding fairseq's gha
* adding py3.9, removing submodules
* yaml linting

* [test]: test with py39 + torch 1.8 nightly

* version fix

* more fix

* fix version function for nightly version

* fix torch_pg build

* invalidate cache

* separate benchmark requirements

* comment

* fixed mypy

* fixed a test

* [test]: test adascale with oss

* minor fix

* add a small comment

* refactor: moved find_tensor_by_shape

* refactor: move test golden data into its own module

* refactor: simplied the train function

* refactor: added comments as suggested

Also, we can save time by only running unittests once instead of
twice (with and without coverage).

* removing the torch util altogether, broken on 1.7.1

* docfix

* fix for torch dist broadcast failing on dummy object

* fix broken link

* remove link

* add direct link

* add direct link

* add direct link

* add direct link

* add direct link

* add direct link
Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>