GitLab

* [zero] add strict ddp mode for chunk init

* [gemini] update gpt example

* [gemini] polish code

* [testing] remove code

* [gemini] make more robust

* [zero] add L2 gradient clipping

* [testing] add MlpModel

* [zero] add unit test for grad clipping

* fix atol

* [gemini] fix init bugs for modules

* fix bugs

* [Gemini] more tests for Gemini

* polish code

* [hotfix] fix zero's incompatibility with checkpoint in torch-1.12

* [zero] add cpu shard init

* [zero] add tiny example test

* [colo_tensor] fix bugs for torch-1.11

* add chunk manager init function

* fix unit tests

* add comment

* add flush=True

* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.

This reverts commit 5be118f4.