GitLab

remove author related info

use magatron's init method for ffn

Laekov/accfix

Simple mlp

i.e., expand bias using torch.repeat_interleave directly

the true bias has been added in FMoeLinear

move away layernorm/residual/dropout

Reproducibility

although order of dp and relu doesn't matter

Test Transformer-XL

activation passed by func can not be saved