GitLab

* gather llama logits

* fix

Co-authored-by: binmakeswell <binmakeswell@gmail.com>

* [llama] refactor inference example to fit sft

* [llama] fix training script to fit gemini

* [llama] fix inference script

[llama] support npu for Colossal-LLaMA-2

* [moe] add mixtral block for single expert

* [moe] mixtral block fwd support uneven ep

* [moe] mixtral block bwd support uneven ep

* [moe] add mixtral moe layer

* [moe] simplify replace

* [meo] support save sharded mixtral

* [meo] support load sharded mixtral

* [meo] support save sharded optim

* [meo] integrate moe manager into plug

* [meo] fix optimizer load

* [meo] fix mixtral layer

* [moe] top2 allow uneven input

* [moe] update capacity computing

* [moe] remove debug info

* [moe] update capacity computing

* [moe] update capacity computing

* [llama] fix memory issue

* [llama] add comment

* [llama] update training script

* [doc] polish docstr

* [plugin] refactor prepare dataloader

* [plugin] update train script

* [gemini] fix param op hook when output is tuple

* [gemini] fix param op hook

* fix: remove unnecessary assert

* test: add more 3d plugin tests

* fix: add warning

* [checkpointio] fix hybrid parallel optim checkpoint

* [extension] fix cuda extension

* [checkpointio] fix gemini optimizer checkpoint

* polish code

* fix script

* fix script

* fix chat nan

* fix chat nan