GitLab

* [WIP] stateful tensor manager

* add eviction strategy

* polish code

* polish code

* polish comment

* add unit test

* fix sampler bug

* polish code

* fix max sampling cnt resetting bug

* fix sampler bug

* polish code

* fix bug

* fix unit test
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>

* adapt model weight initialization for methods in Pytorch nn.init

* support existing sharded and unsharded parameters in zero

* add unitest for moe-zero model init

* polish moe gradient handler

* hijack p.grad in sharded model

* polish comments

* polish comments

only process module's own parameters in Zero context

add zero hooks for all modules that contrain parameters

gather parameters only belonging to module itself

* sharded model supports reuse fp16 shard

* rename variable

* polish code

* polish code

* polish code

* sharded optim support hybrid cpu adam

* update unit test

* polish docstring

* [zero] polish sharded param name

* polish code

* polish

* polish code

* polish

* polsih

* polish

* polish code

* shard strategy receive pg in shard() / gather()

* update zero engine

* polish code

* Revert "polish code"

This reverts commit 8cf7ff08.

* Revert "rename variables"

This reverts commit e99af94a.

* Revert "remove surplus imports"

This reverts commit 46add4a5.

* Revert "update sharded optim and fix zero init ctx"

This reverts commit 57567ee7.