GitLab

- fixes #1107
Co-authored-by: Min Xu <min.xu.public@gmail.com>

Just use `torch.inf`, as torch._six is gone after https://github.com/pytorch/pytorch/pull/94709

* [fix] typo in wikitext2_data.py

* [fix] typo and code duplication in fsdp.py

heirarchy -> hierarchy

* [test] ci py 3.11 tests
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* fixed setup.py

* fixed ci config

* fixed ci config's python 3.11 version

* fixed torch installs on cpu

* update pygit2 for 3.11

* we don't run benchmark on cpu, so no need to install the benchmark reqs

* update torch install

* try to install torchvision

* numpy version 311

* fix cpu test dependency installation

* pip git install cmd fix

* bypass some tests in 3.11. failure due to packages they use haven't been updated for 3.11 yet
Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

* Fix gradient accumulation

Add ``is_scaled_loss`` flag to support both scaled / unscaled loss
Fix ``test_grad_accum`` and``test_set_num_gradients_to_accumulate``

* Add a method to scale grad for grad_accum using unscaled loss

- Revert the changes in `step` method
- Add a method `scale_grad_by_num_grads_to_accum`to handle gradient accumulation using unscaled loss more explicitly
- Add gradient tests

* Implement _compute_corr_mean_between_grads

* Improve tests and comments

* Use ubuntu-20.04 instead of latest

Use ubuntu-20.04 to fix the `arch x64 not found` issue
[Version 3.10 with arch x64 not found actions/setup-python#401](https://github.com/actions/setup-python/issues/401)

* Switch flake8 from gitlab to github

Flake8 was moved to Github
See discussions https://www.reddit.com/r/Python/comments/yvfww8/flake8_took_down_the_gitlab_repository_in_favor/

* Fix scikit-learn package

* Update PyTorch v...

Co-authored-by: Min Xu <min.xu.public@gmail.com>

* Fix gradient accumulation

- Add ``is_scaled_loss`` flag to support both scaled / unscaled loss
- Add a method `scale_grad_by_num_grads_to_accum`to handle gradient accumulation using unscaled loss more explicitly
- Fix ``test_grad_accum`` and``test_set_num_gradients_to_accumulate``
- Add tests for gradient

Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

* simlificed the readme

* clean up ssd offload

* try to fix readthedocs
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* tmp

* test again

* test again

* add new test

* clean up

* add test file to the testlist

* more comments

* add changelog
Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

- Fixes from Benjamin.

Original commit msg:
  - Fixes #1041. I just had a minute or two, hoping that it's enough :)
Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [fix] better handling non-flatten in FSDP

- see the detailed comment about that backward firing case
- also minor debugging help in FSDP
- also minor fix in FPW's state dict

* [feat] disallow reset_parameters by default

* [feat] adding fsdp_instances API - useful in check wrapping by user code

* [fix] one line fix but more than a day of debugging

* fixed the case of loading combined check with empty fsdp instances

* fixed another bug around state loading the root/nonroot module full param caching due to not resharding after forward

* [feat] support .half and .float better

* fixed a bug in gather optim state losses extra keys from the original state_dict

* fixed a test failure in mixed precision

* fixed another bug affecting no_sync grad acc

* fixed a bug and a test in fsdp optim state

* fixed another corner case

* added a comment

* skip ssd offload tests

* skip fsdp one for ssd overload
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [fix] don't import ProcessGroup eagerly

- move the import into typing since it is only used for type checking
- fixes #1057

* more fixes

* one more

* tested at least
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [bug] fix optim state gather when there is empty FSDP instances

* fixes an anssert and a test bug

* [minor] [FSDP] add a better for pure fp16

* [minor] [wrap] add a flag to help fsdp pure fp16 wrapping

* [feat] add random_sparse_mask api

* correct test skip
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [fix]: add a context for supporting state_dict from a non-FSDP parent module

* formatting
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [feat] support sst disabled and dst disabled cases

* added tests
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* update nightly version

* update wgit to use numpy for load/store

- this is introduced with new nightly torch version, which made torch.save() not
  producing deterministic bytes
- this make tensor<->numpy conversion and then do the save/load to avoid that issues.

* fixed tests
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* added a profiling class

* no more type ignore after merging main

* fixed a int/round bug

* add unit tests

* skip if no cuda for a test
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* update examples and comment

* fixed issue with fft/ifft only doing the last dim

* fixed a int/round bug; fixed tests

* add cuda tests

* add atol and rtol

* skip cuda test correctly
Co-authored-by: Min Xu <min.xu.public@gmail.com>

* [Feat] implements lossy_compress with tests

1. Implements a method lossy_compress that takes in a dense tensor and returns a reconstruction with sst and dst, and optionally with sparsity.

[Feat] Implements dense_sst_to_dst and sst_dst_to_dense methods and adds tests

1. Implements the dense_sst_to_dst and sst_dst_to_dense method.
2. Adds tests for perfect reconstruction with all top-k across different dims.
3. Adds tests for the two new methods.