GitLab

* refactor colo-tensor and update linear op

* polish code

* polish code

* update ops and unit tests

* update unit tests

* polish code

* rename dist_spec module

* polish code

* polish code

* remove unneeded import

* fix pipelinable

* derive compute pattern from dist spec

* polish code

* add dist spec

* update linear op

* polish code

* polish code

* update embedding op

* polish unit tests

* polish unit tests

* polish comments

* polish code

* add test_dist_spec_mgr

* polish code

* refactor folder structure

* polish unit tests

* add get_process_group() for TensorSpec

* polish code

* add optimizer to bert test

* polish

* change torch.Parameter to ColoParameter

* fix post assignment for init context

* polish

* polish

* simplify ColoModulize

* simplify ColoModulize

* polish

* polish

* hijack addmm for colo tensor

* fix bugs

* polish unit test

* polish comments

* add from_pretrained support and test

* polish

* polish

* polish

* polish

* add 1d vocab loss

* polish

* polish test_model

* polish

* add base bert test

* Add bert test

* polish

* remove test_bert

* polish

* [Tensor] activation is an attr of ColoTensor

* [Tensor] add optimizer

* only detach parameters in context

* polish code

* refine linear and add function to ColoTensor

* add gather for layernorm

* polish

* polish

* [Tensor] make ColoTensor more robust for getattr

* polish

* polish

* add lru cache

* polish code

* update unit test

* fix sharded optim

* [Tensor] add some function to ColoTensor

* torch.allclose

* rm torch.add

* [tensor] customized op returns ColoTensor

* polish

* polish code