GitLab

- this function is being removed in pytorch
- we only need to call it in case we are working with older pytorch
Co-authored-by: Min Xu <min.xu@acm.org>

* relax checking root condition

* formatting

* add unittest

* add unittest to ci test list

* isort for import of unittest

* format black .

* move test to list 1

* add skip no cuda

* black and isort

* [fix] mypy and flaky test

- CI didn't seem to catch this or maybe I merged incorrectly yesterday
- this should fix the mypy error on master
- also updated a test that seems to be flaky due to tcp port conflict

* another flaky test, hopefully more determinism helps

* CR

* skip 1.6

* fix

* minor
Co-authored-by: Min Xu <min.xu@acm.org>

* Changing FSDP init to by pass pg validation for freshly minted pgs inside of init.

* Addressing Min's review comments.

* Changing logging in init to debug from info

* Changing logging in init to debug from info
Co-authored-by: Giri Anantharaman <giriman@devfair0439.h2.fair>

* Changing FSDP init to by pass pg validation for freshly minted pgs inside of init.

* Addressing Min's review comments.
Co-authored-by: Giri Anantharaman <giriman@devfair0439.h2.fair>

* [chore] 0.3.5 release

* address comment
Co-authored-by: Min Xu <min.xu@acm.org>

* FSDP: fixing training with freezing weights

- an assert is changed to catch this case correctly
- unit test added (based on Quentin's test code) for this case and
  compare DDP and FSDP

fixes: #610

* added test file to list 1

* Use better and simpler code as suggested by Myle

* testing both methods of freezing as well
Co-authored-by: Min Xu <min.xu@acm.org>

[fix] Revert change that removed the option to run OffloadModel with out activation checkpointing. (#608)

* revert change made

* add tests and revert sync shard changes

* add tests

* remove file checked in by error

* inine var

* fix lint errors

* add checkpoint activation

* fix mypy

* use a bigger model

* modify tests for now

* resolve conflicts
Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>

* modify doc string

* add offload docs

* add tutorial

* remove print

* remove print statement

* modify import

* modify constants

* modify README and add Offload symbol

* fix lint

* smaller mods

* lint errors

* Update README.md

added the references at the bottom of the readme

* address comments

* doc changes

* add blank line
Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair>
Co-authored-by: Vittorio Caggiano <caggiano@gmail.com>

replacing multip-process pipe implementation with more flexible one

Initial implementation of proposal pytorch/pytorch#55256

* Adding a unit test which checks for multiple FW passes on the same block
* Adding an embedding table, but still no problem to show for it

* Properly handle .train() and .eval() modes
* showing that the unit test works, now fixed
* code review

* debugging

* debugging activation issue

* fix activation loading

* remove changes used for testing

* remove comment

* add model

* add offload regression benchmarks

* add golden data

* remove mp pipe benchmark

* fix lint

* remove rank

* add check for model type

* lint errors

* making APIs more private
* linting

* fixing given torchvision's change

This test is flaky for torch >= 1.8.0.

NCCL all_to_all is now supported in PyTorch (since v1.8.0)

Fixes: #548

- releasing 0.3.3
- I need it in vissl for the auto_wrap_bn change

* add record_function support

* add more record_function cutpoints

* add more record_function cutpoints

* lint errors

* make string ids more specific