GitLab

* add set_params_to_ignore for ColoDDP

* polish code

* fix zero hook v2

* add unit test

* polish docstr

* [pipeline] refactor the pipeline module

* polish code

* [context] support lazy init of module

* polish code

* [test] skip tests when not enough GPUs are detected

* polish code

* polish code

* [test] ignore 8 gpu test

* polish code

* polish workflow

* polish workflow

* refactor StatefulTensor, tensor utilities

* add unitest for GeminiMemoryManager

* [CLI] add CLI launcher

* Revert "[CLI] add CLI launcher"

This reverts commit df7e6506d4500af6a9220ef7fe4d3c7b1daebd4c.

* [pipeline]add module lazy init feature to support large model initization.

* [pipeline]add to_layer_list and partition method to support arbitrary non-pp model

* refactor the module structure

* polish

* [pipelinable]add unit test for pipelinable

* polish

* polish

* Fix CodeFactor issues.

* [test] refactored with the new rerun decorator

* polish test case

* remove hybrid adam in test_moe_zero_optim

* fix activation checkpointing and its unitest

* [zero] polish sharded param name

* polish code

* polish

* polish code

* polish

* polsih

* polish

* Added tensor detector

* Added the - states

* Allowed change include_cpu when detect()

* refactored test with component func

* fixed bug

* Added activation offload

* Fixed the import bug, used the pytest

This reverts commit bef05489b642385c80e59fe757d598efd1752ecf.

* add zero1 (#209)

* add zero1

* add test zero1

* update zero stage 1 develop (#212)

* Implement naive zero3 (#240)

* naive zero3 works well

* add zero3 param manager

* add TODOs in comments

* add gather full param ctx

* fix sub module streams

* add offload

* fix bugs of hook and add unit tests

* fix bugs of hook and add unit tests (#252)

* add gather full param ctx

* fix sub module streams

* add offload

* fix bugs of hook and add unit tests

* polish code and add state dict hook

* fix bug

* update unit test

* refactor reconstructed zero code

* clip_grad support zero3 and add unit test

* add unit test for Zero3ParameterManager

* [WIP] initialize the shard param class

* [WIP] Yet another sharded model implementation (#274)

* [WIP] initialize the shard param class

* [WIP] Yes another implementation of shardModel. Using a better hook method.

* torch.concat -> torch.cat

* fix test_zero_level_1.py::test_zero_level_1 unitest

* remove deepspeed implementation and refactor for the reconstructed zero module

* polish zero dp unittests
Co-authored-by: ver217 <lhx0217@gmail.com>
Co-authored-by: Frank Lee <somerlee.9@gmail.com>

* optimized 1d layer apis; reorganized nn.layer modules; fixed tests

* fixed 2.5d runtime issue

* reworked split batch, now called in trainer.schedule.load_batch
Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>