GitLab

Fix deadlock when get_num_microbatches() < pipeline-parallel size

See merge request ADLR/megatron-lm!243

Fix deadlock when get_num_microbatches() < pipeline-parallel size (don't try to measure pipeline stall)

Support Torch DDP for single-stage, num_microbatches() > 1

See merge request ADLR/megatron-lm!240

Don't import deprecated model from realm_model which is broken.

See merge request ADLR/megatron-lm!241

fix warning condition

See merge request ADLR/megatron-lm!238

Storing and loading fingerprints of in deduplication

See merge request ADLR/megatron-lm!236

Fix interleaved schedule assertion

See merge request ADLR/megatron-lm!237

Bug Jaccard similarity and filtering n-grams

See merge request ADLR/megatron-lm!234

ICT Retriever

See merge request ADLR/megatron-lm!235

Merge ICT Retriever with Main branch

See merge request ADLR/megatron-lm!227

Interleaved pipelined execution and refactored schedule / communication code

See merge request ADLR/megatron-lm!189

- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions

Improve handling of rng states in checkpoints.

See merge request ADLR/megatron-lm!231