GitLab

Changing the interface to the lm eval harness and fixing bugs caused by misunderstanding out_seq_length

allreduce overlap with wgrad gemm

change custom delay to dummy add

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

added across rank sync for checkpoint iteration laod, fixed type for timing, and validation iterations

Some tests expect a clean model parallel slate and complain if a previous test left something behind; this change clears more variables that the tests complain about.