GitLab

* [ddp] add is_ddp_ignored

[ddp] rename to is_ddp_ignored

* [zero] fix state_dict and load_state_dict

* fix bugs

* [zero] update unit test for ZeroDDP

* [gemini] polish code

* [testing] remove code

* [gemini] make more robust

* [Gemini] more tests for Gemini

* polish code

* add chunk manager init function

* fix unit tests

* add comment

* add flush=True

* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.

This reverts commit 5be118f4.