[moe] merge moe into main (#4978)

* update moe module * support openmoe

[moe] merge moe into main (#4978)
* update moe module * support openmoe
dc003c30 · Xuanlei Zhao · GitHub · 8993c8a8 · dc003c30 · dc003c30
Unverified Commit dc003c30 authored Nov 02, 2023 by Xuanlei Zhao Committed by GitHub Nov 02, 2023
5 changed files
--- a/tests/test_moe/test_moe_router.py
+++ b/tests/test_moe/test_moe_router.py
--- a/tests/test_moe/test_moe_zero_fwd_bwd.py
+++ b/tests/test_moe/test_moe_zero_fwd_bwd.py
--- a/tests/test_moe/test_moe_zero_init.py
+++ b/tests/test_moe/test_moe_zero_init.py
--- a/tests/test_moe/test_moe_zero_model.py
+++ b/tests/test_moe/test_moe_zero_model.py
--- a/tests/test_moe/test_moe_zero_optim.py
+++ b/tests/test_moe/test_moe_zero_optim.py