GitLab

Run evaluation pass before checkpoint is saved

See merge request ADLR/megatron-lm!196

Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint

Option to run residual connections in fp32

See merge request ADLR/megatron-lm!195

Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

Pipeline parallelism and inter-layer model parallelism implementation

See merge request ADLR/megatron-lm!159