GitLab

* [autoparallel] resnet block runtime apply

* seperate buffer and parameter in MemoryCost

* polish code

* add comments and todos

* fix test issue

* fixes memory leak when paramter is in fp16 in ZeroDDP init.
* bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release.
* adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.

* [autoparallel] refactored the autoparallel module for organization

* polish code

* [autoparallel] adapt runtime passes v2

* polish code

polish code

* [autoparallel] collated all deprecated files

* polish code

* [autoparallel] adapt solver and CostGraph with new handler

* fix test issue

* [autoparallel] add output handler and placeholder handler

* Delete test_solver_with_resnet.py

* fix test bugs

* [autoparallel] add pooling handler

* polish code

* where generator

* [autoparallel] where_handler_v2

* [fx/profiler] modify data_ptr into uuid for all tensors.

* [fx] modify uuid.

* [fx/profiler] tune performance on GPT-2.

* [fx] updates.

* [fx] debug.

* [fx] debug.

* [fx] cuda.

* [autoparallel] add following node generator

* polish code

* polish code

* update name of arguments

* [autoparallel] add layer norm handler v2

* polish code

* polish code

* [autoparallel] add rotor c version

* [fx] remove metainfoprop in rotor solver

* [autoparallel] modify C
 code format

* [autoparallel] remove build.py

* [autoparallel] fix C extension build

* [autoparallel] add C solver consistency test

* [autoparallel] remove some unused imports

* [autoparallel] refactor rotor solver code

* [autoparallel] replace print with colossalai logger

* [autoparallel] ranks fixed

* [autoparallel] added utils for broadcast operation

* polish code

[pipeline/pytree] add pytree to process args and kwargs | provide `data_process_func` to process args and kwargs after forward (#1642)

* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera

* [pipeline/chimera] test chimera | fix bug of initializing

* [pipeline/pytree] add pytree to process args and kwargs | provide  to process args and kwargs after forward

* [autoparallel] change the following nodes strategies generation logic

* fix unit test

* [autoparallel] where handler

* fix unit test

* [fx] fix offload codegen test

* [fx] modify typing

* [fix] fixed the collective pattern name for consistency

* polish code

This reverts commit 5be118f4.