GitLab

This is no longer needed since isort's version is 5.10

Also fix black version to 22.3.0 to fix issue with click
dependency.

Update files that now fail with new version of black {a = 2 ** 4} ->
{a = 2**4}

* copyright headers

* isort and pyproject.toml

* precommit and requirement for isort-seed-config

* mypy

* dummy change

* numpy version for pre-commit

* fix mypy issue caused by numpy
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [skip ci] first commit

* [skip ci] gradient scaler example

* [skip ci] adding feed forward toy example

* [skip ci] adding types

* [skip ci] adding backward hook

* [skip ci] update

* [skip ci] working feed forward example

* [skip ci] working feed forward example

* [skip ci] use named_modules instead of named_children

* [skip ci] adding new file

* [skip ci] clean up

* [skip ci] implement unscale function

* [skip ci] implement unscale function

* [skip ci] removing old file

* [skip ci] removing some more old files

* [skip ci] making unscale function generic

* [skip ci] adding test for vision model

* [skip ci] adding identity layer

* [skip ci] cleanup files

* [skip ci] refactoring

* [skip ci] more refactoring

* [skip ci] added functionality to update scale

* [skip ci] data loader clean up

* [skip ci] implemented inf checks and update scale functions

* [skip ci]code clean up. added test with autocast. does not work atm

* adding documentation

* adding dependency in requirements-dev.txt

* updating pytorch nightly version

* updating changelog

* adding is_cuda_available to test_vision_model

* set same timeout on cpu and gpu

* reverting cpu timeout, skip vision test on cpu

* addressing comments, fixing vision test

* unscale uses in-place matmul

* some more cleanup

* Add MOE to lm benchmarks

* linter

* Fix source / target

* address comments

* address comments

* address comments

* add circleci

* fix circleci

* precommit

* [fix]: fix eval for shared weight FSDP

* fixing optim state saving

* add changelog

* reformat with newer local isort

* update test

* avoid computing reference state unless we are testing training

* added optim_state test

* make mypy happy

* move tests; maybe we need to CUDA memory related tests in the first of the lists
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* fixed lint issues

* remove unused print statements

* add changelog entry

* [skip ci] fix lint errors

* adding pre-commit files

* applying pre-commit to all files

* adding no-strict-optional argument to mypy in circle ci config

* fix typo

* updating python versions

* [skip ci] remove extra args

* adding python 3.9

* [skip ci] set pre-commit version in requirements-dev.txt

* set CACHE_VERSION

* move linters from circleci to github actions

* update python version

* update python version in benchmarks_2

* moving to python 3.9.7

Add SlowMo Distributed Data Parallel for clusters with slow interconnects
Co-authored-by: Vinayak Tantia <tantia.vinayak1@gmail.com>

Pre-commit hook fails when run on all files for three reasons:
(see trace below)

1. Trailing whitespace on multiple files
2. mypy fails to load numpy and then subsequently fails to load
LazyModule from pipe.py
3. isort sees issues with known_third_party packages

```
> pre-commit run --all-files

Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing docs/source/conf.py
Fixing fairscale/experimental/nn/auto_shard.py
Fixing docs/source/deep_dive/activation_checkpointing.rst
Fixing docs/source/tutorials/pipe.rst
Fixing docs/source/installation_instructions.rst
Fixing docs/source/deep_dive/pipeline_parallelism.rst
Fixing docs/source/tutorials/activation_checkpointing.rst
Fixing docs/source/tutorials/offload_model.rst
Fixing docs/source/deep_dive/oss_sdp_fsdp.rst
Fixing docs/source/what_is_fairscale.rst
Fixing CHANGELOG.md
Fixing fairscale/experimental/nn/offload.py
Fixing docs/source/index.rst
Fixing docs/source/deep_dive/adascale.rst
Fixing README.md
Fixing docs/source/tutorials/oss.rst
Fixing docs/source/deep_dive/offload.rst

Check python ast.........................................................Passed
Check for merge conflicts................................................Passed
Don't commit to branch...................................................Passed
Check for added large files..............................................Passed
Fix End of Files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing requirements.txt
Fixing docs/source/getting_started.rst
Fixing docs/source/installation_instructions.rst
Fixing codecov.yml
Fixing docs/source/deep_dive/adascale.rst
Fixing docs/source/tutorials/oss.rst
Fixing docs/source/deep_dive/offload.rst

black....................................................................Passed
flake8...................................................................Passed
seed isort known_third_party.............................................Failed
- hook id: seed-isort-config
- exit code: 1
- files were modified by this hook
isort....................................................................Passed
mypy.....................................................................Failed
- hook id: mypy
- exit code: 2

setup.cfg:45: error: Error importing plugin 'numpy.typing.mypy_plugin': No module named 'numpy'
Found 1 error in 1 file (checked 197 source files)
```

* POC, testing against the DDP comm hook when available
* docs, adding a reference to DDP's compress hook
* updating changelog, prep for v0.1.8 release

authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>

* adding fairseq's gha
* adding py3.9, removing submodules
* yaml linting

* fix for torch dist broadcast failing on dummy object

* tentatively fixing the cpu version of circleci jobs, now pipe tests are the last ones standing
* fixing oss backcompat, trying to fix rpc in old pytorch also
* fixing the file based init in torch 1.5

* add additional unit test
* support model parallelism in oss

* lintfixes

* come on black

* Update tutorial_pipe_multiprocess.py

make RANK global like the other tutorials
Co-authored-by: Vittorio Caggiano <caggiano@gmail.com>

Adds support for:
* Reused layers (e.g. for weight sharing)
* Lazily-constructed layers
* Single-process control via PipeRPCWrapper
* PipelineStyle.AsyncScheudle, which lays the foundation for asynchronous pipeline work by introducing an event loop for each rank/worker to process either activations or gradients as they arrive

Also added examples for multi-process and PipeRPCWrapper