GitLab

* [hotfix] fix zero's incompatibility with checkpoint in torch-1.12

* [zero] add cpu shard init

* [zero] add tiny example test

* [colo_tensor] fix bugs for torch-1.11

* add chunk manager init function

* fix unit tests

* add comment

* add flush=True

* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.

This reverts commit 5be118f4.

* add lru cache

* polish code

* update unit test

* fix sharded optim

* refactor StatefulTensor, tensor utilities

* add unitest for GeminiMemoryManager

* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e28.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish