GitLab

Only create task heads on last pipeline stage.

See merge request ADLR/megatron-lm!202

Use batched send and recv instead of torch.distributed.ring_exchange()

See merge request ADLR/megatron-lm!198

Run evaluation pass before checkpoint is saved

See merge request ADLR/megatron-lm!196

Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint

Option to run residual connections in fp32

See merge request ADLR/megatron-lm!195

Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

Pipeline parallelism and inter-layer model parallelism implementation

See merge request ADLR/megatron-lm!159