GitLab

remove author related info

use magatron's init method for ffn

Laekov/accfix

Simple mlp

i.e., expand bias using torch.repeat_interleave directly

the true bias has been added in FMoeLinear

move away layernorm/residual/dropout

Reproducibility

although order of dp and relu doesn't matter

Test Transformer-XL

activation passed by func can not be saved

follow suggestions from https://github.com/pytorch/pytorch/issues/36035#issuecomment-770960405