GitLab

Co-authored-by: henryqin1997 <henryqin1997@gamil.com>

[NFC] polish colossalai/auto_parallel/tensor_shard/node_handler/getitem_handler.py code style (#2289)

* add meta_data_computing

* add checkpoint_annotation

* rename proxy.data to proxy.meta_data and add bias addition pass

* polish code

* delete meta_prop_pass invoke and rename ori_node to orig_node

* add TracerType

* unify meta data computing

* delete TracerType

* handle setitem operation

* operator.setitem

* [amp] add gradient clipping in unit tests

* fix bugs

[autockpt] provide option for activation checkpoint search in SPMD solver

* [autoparallel] align the data_ptr with the old version of auto activation checkpoint pipeline

* [autoparallel] using fwd_time and bwd_time instead of fwd_flop and bwd_flop

* [autoparallel] specifycomm nodes' memory cost in construct chain

* [autoparallel] fix wrong runtime apply calculation

* [autoparallel] fix wrong runtime apply calculation

* [autoparallel] fix wrong runtime apply calculation

* [autoparallel] bypass metainfo when available and modify BCAST_FUNC_OP

* [autockpt] make it work.

* [autockpt] linearize / merge shape-consistency nodes.

* [autockpt] considering parameter and optimizer weights.

* [hotfix] pass a parameter.

* add benchmark

* merge common func

* add total and avg tflops
Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>

* [autoparallel] align the data_ptr with the old version of auto activation checkpoint pipeline

* [autoparallel] using fwd_time and bwd_time instead of fwd_flop and bwd_flop

* [autoparallel] specifycomm nodes' memory cost in construct chain

* [autoparallel] fix wrong runtime apply calculation

* [autoparallel] fix wrong runtime apply calculation

* [autoparallel] fix wrong runtime apply calculation

* [autockpt] make it work.

* [autockpt] linearize / merge shape-consistency nodes.

* [autockpt] considering parameter and optimizer weights.

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>

* [autockpt] make it work.

* [autockpt] linearize / merge shape-consistency nodes.

* [autoparallel] gpt2 autoparallel examples

* polish code

* polish code

* move to cpu to avoid dead lock

* get output by offsets
Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>

* [autoparallel] align the data_ptr with the old version of auto activation checkpoint pipeline

* [autoparallel] using fwd_time and bwd_time instead of fwd_flop and bwd_flop

* [autoparallel] specifycomm nodes' memory cost in construct chain

[autoparallel] align the data_ptr with the old version of auto activation checkpoint pipeline (#2261)

* [autoparallel] hook node meta on graph nodes for checkpoint solver

* [autoparallel] polish code

* [autoparallel] restore some node handlers

* colossalai/auto_parallel/passes/meta_info_prop.py

* [autoparallel] remove some unused import

* [autoparallel] hook bwd_mem_out

* [autoparallel] patch torch.flatten