GitLab

* [zero] add strict ddp mode

* [polish] add comments for strict ddp mode

* [zero] fix test error

* [gemini] polish code

* [testing] remove code

* [gemini] make more robust

* [testing] add beit model

* [beit] fix bugs

* [beit] fix bugs

* [testing] fix bugs

* Add mix-gather

* Add comments

* Add comments

* Polish comments

* Change the global rank assumption

* Add tests

* Add two-step tests

* Fix 10 and 01

* Skip test becasue the number of GPUs

* [autoparallel] shard param and buffer as expected

* fix unit test issue

* [autoparallel] handled illegal sharding strategy

* polish code

* add chunk manager init function

* fix unit tests

* add comment

* add flush=True

* [fix] fixed the collective pattern name for consistency

* polish code

This reverts commit 5be118f4.

* [tensor] use communication autograd func

* change all to all comm spec info

* rename pattern and distinguish fwd/bwd

* polish code

* [tensor] support runtime ShardingSpec apply

* polish code

* polish code

* [tensor] shape consistency output transform path and communication cost

* polish code

* [tensor] added linear implementation for the new sharding spec

* polish code

* [tensor] add shape consistency feature to supportauto sharding spec transform.

* [tensor] remove unused argument in simulator, add doc string for target pair.

[colotensor] add megatron initialization for gpt2

* process group supports getting ranks in group

* chunk mgr receives a process group

* update unit test

* fix unit tests