GitLab

* fix sharded optim zero grad

* polish comments

* support existing sharded and unsharded parameters in zero

* add unitest for moe-zero model init

* polish moe gradient handler

Change the clang-format style to google style

* hijack p.grad in sharded model

* polish comments

* polish comments

* add memory trainer hook

* fix bug

* add memory trainer hook

* fix import bug

* fix import bug

* add trainer hook

* fix #370 git log bug

* modify `to_tensorboard` function to support better output

* remove useless output

* change the name of `MemProfiler`

* complete memory profiler

* replace error with warning

* finish trainer hook

* modify interface of MemProfiler

* modify `__init__.py` in profiler

* remove unnecessary pass statement

* add usage to doc string

* add usage to trainer hook

* new location to store temp data file

* optimize grad offload

* polish code

* polish code

only process module's own parameters in Zero context

add zero hooks for all modules that contrain parameters

gather parameters only belonging to module itself

* [zero] fix grad offload

* polish code

* [zero]added hybrid adam, removed loss scale of adam

* remove useless code