GitLab

* adapt model weight initialization for methods in Pytorch nn.init

* remove hybrid adam in test_moe_zero_optim

* fix activation checkpointing and its unitest

* add cpu adam counter for all cpu adam

* fixed updating error in adam kernel

* support existing sharded and unsharded parameters in zero

* add unitest for moe-zero model init

* polish moe gradient handler

* hijack p.grad in sharded model

* polish comments

* polish comments

only process module's own parameters in Zero context

add zero hooks for all modules that contrain parameters

gather parameters only belonging to module itself

* [zero]added hybrid adam, removed loss scale of adam

* remove useless code

* sharded model supports reuse fp16 shard

* rename variable

* polish code

* polish code

* polish code

* sharded optim support hybrid cpu adam

* update unit test

* polish docstring

* [zero] polish sharded param name

* polish code

* polish

* polish code

* polish

* polsih

* polish