GitLab

* [NFC] polish colossalai/fx/profiler/experimental/profiler_module/embedding.py code style

* [NFC] polish colossalai/communication/utils.py code style

---------
Co-authored-by: Minghao Huang <huangminghao@luchentech.com>

Co-authored-by: aye42 <aye42@gatech.edu>

Co-authored-by: supercooledith <893754954@qq.com>

* revise shardformer readme (#4246)

* [example] add llama pretraining (#4257)

* [NFC] polish colossalai/communication/p2p.py code style

---------
Co-authored-by: Jianghai <72591262+CjhHa1@users.noreply.github.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>
Co-authored-by: Qianran Ma <qianranm@luchentech.com>

* [NFC] polish colossalai/booster/mixed_precision/mixed_precision_base.py code style

* sharded optimizer checkpoint for gemini plugin

* modify test to reduce testing time

* update doc

* fix bug when keep_gatherd is true under GeminiPlugin

* [lazy] support init on cuda

* [test] update lazy init test

* [test] fix transformer version

* added softmax kernel

* added qkv_kernel

* added ops

* adding tests

* upload tets

* fix tests

* debugging

* debugging tests

* debugging

* added

* fixed errors

* added softmax kernel

* clean codes

* added tests

* update tests

* update tests

* added attention

* add

* fixed pytest checking

* add cuda check

* fix cuda version

* fix typo

Co-authored-by: github-actions <github-actions@github.com>

* [docker] fixed ninja build command

* polish code

* [checkpointio] unsharded optimizer checkpoint for Gemini plugin

* [checkpointio] unsharded optimizer checkpoint for Gemini using all_gather

Co-authored-by: github-actions <github-actions@github.com>

* [shardformer] made tensor parallelism configurable

* polish code

* [shardformer] refactored some doc and api

* polish code

* [shardformer] add benchmark of shardformer

* [shardformer] add benchmark of shardformer