GitLab

allreduce overlap with wgrad gemm

change custom delay to dummy add

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

added across rank sync for checkpoint iteration laod, fixed type for timing, and validation iterations

Some tests expect a clean model parallel slate and complain if a previous test left something behind; this change clears more variables that the tests complain about.