GitLab

* [moe] add checkpoint for moe models

* [hotfix] fix bugs in unit test

* remove forced FP32 modules

* correct no_shard-contexts' positions

* [test] refactored with the new rerun decorator

* polish test case

* add tensor placement policies

* polish comments

* polish comments

* update moe unit tests

* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters

* adapt model weight initialization for methods in Pytorch nn.init

* remove hybrid adam in test_moe_zero_optim

* fix activation checkpointing and its unitest

* add cpu adam counter for all cpu adam

* fixed updating error in adam kernel

* support existing sharded and unsharded parameters in zero

* add unitest for moe-zero model init

* polish moe gradient handler

Decreased moe tests;

Added FFNExperts and ViTMoE model