GitLab

* fixed bugs of assigning grad states to non leaf nodes

* use detach()

* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters

* refactor memstats collector

* fix disposable

* polish code

* add PaLM link

* [WIP] stateful tensor manager

* add eviction strategy

* polish code

* polish code

* polish comment

* add unit test

* fix sampler bug

* polish code

* fix max sampling cnt resetting bug

* fix sampler bug

* polish code

* fix bug

* fix unit test
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>

Co-authored-by: github-actions <github-actions@github.com>

Co-authored-by: github-actions <github-actions@github.com>

* adapt model weight initialization for methods in Pytorch nn.init

* refactor pipeline---put runtime schedule into engine.

* add type hint for schedule Optional[BaseSchedule]

* preprocess schedule during engine initializing

* infer pipeline schedule params from config

[NFC] polish colossalai/context/process_group_initializer/initializer_sequence.py colossalai/context/process_group_initializer initializer_tensor.py  code style (#639)
Co-authored-by: 何晓昕 <cautious@r-236-100-25-172.comp.nus.edu.sg>

[NFC] polish colossalai/kernel/cuda_native/csrc/kernels/dropout_kernels.cu and cross_entropy.cu code style (#634)

Co-authored-by: RichardoLuo <14049555596@qq.com>