GitLab

Fix pytorch compatibility issue !55

* Added default weight initializations to FMoELinear and NoisyGate

* Following torch's naming convention

use single variable for returned value

the old impl raised error "too many values to unpack (expected 1)"

Fix grad of balance loss

* use single variable instead of vector in c functions

* expert count kernel

* remove all lists

* fix old tests

Fixed asynchronous streams in column reduce kernel call

Checkout version number to V0.2.0

Add hidden hidden size args

Adapt balance loss for new gate interface & update patch

Fix bugs to run megatron with gshard gate

Update test_gates.py

Add random routing in gshard gate

mask and experts list

- mask some tensors of tokens for fmoe forward
- pass a list of expert classes to specify what experts in what order want to use

Reconstruct gate and add gshard / switch

Bias improvement #15