GitLab

Update benchmark according to update of gates

Grad clip patch for megatron v2.2

fix fp16 training with balance loss

https://github.com/laekov/fastmoe/issues/63

Remove unnecessary dependencies on comm

Use moe_group instead of world for MoE

Fix typo in readme

Fix pytorch compatibility issue !55

* Added default weight initializations to FMoELinear and NoisyGate

* Following torch's naming convention

use single variable for returned value

the old impl raised error "too many values to unpack (expected 1)"

Fix grad of balance loss

* use single variable instead of vector in c functions

* expert count kernel

* remove all lists

* fix old tests

Fixed asynchronous streams in column reduce kernel call

Checkout version number to V0.2.0

Add hidden hidden size args

Adapt balance loss for new gate interface & update patch

Fix bugs to run megatron with gshard gate

Update test_gates.py

Add random routing in gshard gate