GitLab

Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch

Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size

Allocate tensor in `communicate()` method directly on GPU (instead of allocating on CPU and then moving to GPU)

Prevents data_loader from running out of training examples

Make sure all forward and backward operations are accounted for