GitLab

Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint

Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch

Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size

Allocate tensor in `communicate()` method directly on GPU (instead of allocating on CPU and then moving to GPU)

Prevents data_loader from running out of training examples

Make sure all forward and backward operations are accounted for