GitLab

* support no sync for zero1 plugin

* polish

* polish

* refactor low level zero

* fix zero2 and support cpu offload

* avg gradient and modify unit test

* refactor grad store, support layer drop

* refactor bucket store, support grad accumulation

* fix and update unit test of zero and ddp

* compatible with tp, ga and unit test

* fix memory leak and polish

* add zero layer drop unittest

* polish code

* fix import err in unit test

* support diffenert comm dtype, modify docstring style

* polish code

* test padding and fix

* fix unit test of low level zero

* fix pad recording in bucket store

* support some models

* polish

Co-authored-by: Yuanchen Xu <yuanchen.xu00@gmail.com>

* [NFC] polish colossalai/context/random/__init__.py code style

* [NFC] polish applications/Chat/coati/models/utils.py code style

Co-authored-by: henryqin1997 <henryqin1997@gamil.com>

Co-authored-by: Yuanchen Xu <yuanchen.xu00@gmail.com>

* [NFC] polish colossalai/fx/profiler/experimental/profiler_module/embedding.py code style

* [NFC] polish colossalai/communication/utils.py code style

---------
Co-authored-by: Minghao Huang <huangminghao@luchentech.com>

Co-authored-by: aye42 <aye42@gatech.edu>

Co-authored-by: supercooledith <893754954@qq.com>

* revise shardformer readme (#4246)

* [example] add llama pretraining (#4257)

* [NFC] polish colossalai/communication/p2p.py code style

---------
Co-authored-by: Jianghai <72591262+CjhHa1@users.noreply.github.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>
Co-authored-by: Qianran Ma <qianranm@luchentech.com>

* [NFC] polish colossalai/booster/mixed_precision/mixed_precision_base.py code style

* sharded optimizer checkpoint for gemini plugin

* modify test to reduce testing time

* update doc

* fix bug when keep_gatherd is true under GeminiPlugin

* [lazy] support init on cuda

* [test] update lazy init test

* [test] fix transformer version

* added softmax kernel

* added qkv_kernel

* added ops

* adding tests

* upload tets

* fix tests

* debugging

* debugging tests

* debugging

* added

* fixed errors

* added softmax kernel

* clean codes

* added tests

* update tests

* update tests

* added attention

* add

* fixed pytest checking

* add cuda check

* fix cuda version

* fix typo

Co-authored-by: github-actions <github-actions@github.com>

* [docker] fixed ninja build command

* polish code

* [checkpointio] unsharded optimizer checkpoint for Gemini plugin

* [checkpointio] unsharded optimizer checkpoint for Gemini using all_gather

Co-authored-by: github-actions <github-actions@github.com>