GitLab

* reorgnize colotensor directory

* polish code

* add zero optimizer

* torch ok

* unit test ok

* polish code

* fix bugs

* polish unit test

* polish zero optim

* polish colo ddp v2

* refactor folder structure

* add comment

* polish unit test

* polish zero optim

* polish unit test

* impl chunk manager

* impl param op hook

* add reduce_chunk

* add zero hook v2

* add zero dp

* fix TensorInfo

* impl load balancing when using zero without chunk

* fix zero hook

* polish chunk

* fix bugs

* ddp ok

* zero ok

* polish code

* fix bugs about load balancing

* polish code

* polish code

* add ene-to-end test

* polish code

* polish code

* polish code

* fix typo

* add test_chunk

* fix bugs

* fix bugs

* polish code

* add Parameter inheritance for ColoParameter

* remove tricks

* remove tricks

* polish

* polish

* add Embedding

* Add bert test

* polish

* add check module test

* polish

* polish

* polish

* polish

* add module spec for linear

* polish

* polish

* polish

* impl ColoDDP for ColoTensor

* polish code

* refactor parallel action

* polish unit tests

* polish test_gpt

* update op unit tests

* update test model

* refactor colo-tensor and update linear op

* polish code

* polish code

* update ops and unit tests

* update unit tests

* polish code

* rename dist_spec module

* polish code

* polish code

* remove unneeded import

* fix pipelinable

* derive compute pattern from dist spec

* polish code

* add dist spec

* update linear op

* polish code

* polish code

* update embedding op

* polish unit tests

* polish unit tests

* polish comments

* polish code

* add test_dist_spec_mgr

* polish code

* refactor folder structure

* polish unit tests

* add get_process_group() for TensorSpec

* polish code

* add optimizer to bert test

* polish

* change torch.Parameter to ColoParameter

* fix post assignment for init context

* polish

* polish

* simplify ColoModulize

* simplify ColoModulize

* polish

* polish

* hijack addmm for colo tensor

* fix bugs

* polish unit test

* polish comments

* add from_pretrained support and test

* polish

* polish

* polish

* polish

* add 1d vocab loss

* polish

* polish test_model

* polish

* add base bert test

* Add bert test

* polish

* remove test_bert

* polish

* [Tensor] activation is an attr of ColoTensor

* [Tensor] add optimizer

* only detach parameters in context

* polish code

* refine linear and add function to ColoTensor

* add gather for layernorm

* polish

* polish

* [Tensor] make ColoTensor more robust for getattr

* polish

* polish