GitLab

allreduce overlap with wgrad gemm

change custom delay to dummy add

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

hidden_size % attention_heads == 0 is handled above when dealing with kv_channels.

Adding check for decoder sequence length.