GitLab

* [test] fixed torchrec model test

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code

* [zero] add strict ddp mode for chunk init

* [gemini] update gpt example

* [ddp] add is_ddp_ignored

[ddp] rename to is_ddp_ignored

* [zero] fix state_dict and load_state_dict

* fix bugs

* [zero] update unit test for ZeroDDP

* [gemini] polish code

* [testing] remove code

* [gemini] make more robust

* [testing] add beit model

* [beit] fix bugs

* [beit] fix bugs

* [testing] fix bugs

* [zero] fix error for BEiT models

* [ColoParameter] add unpack operation for tuple arguments

* fix bugs

* fix chunkv2 unit testing

* add assertion for gradient state

* [zero] add L2 gradient clipping

* [testing] add MlpModel

* [zero] add unit test for grad clipping

* fix atol

* [gemini] fix init bugs for modules

* fix bugs

* [Gemini] more tests for Gemini

* polish code

* [testing] fix testing models

* roll back

* [hotfix] fix zero's incompatibility with checkpoint in torch-1.12

* [zero] add cpu shard init

* [zero] add tiny example test

* [colo_tensor] fix bugs for torch-1.11

* add chunk manager init function

* fix unit tests

* add comment

* add flush=True

* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.