GitLab

Interleaved pipelined execution and refactored schedule / communication code

See merge request ADLR/megatron-lm!189

- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions

Improve handling of rng states in checkpoints.

See merge request ADLR/megatron-lm!231

Use torch.cuda.synchronize() right after calling batch_isend_irecv() communication API

See merge request ADLR/megatron-lm!230

conditioning fused kernels

See merge request ADLR/megatron-lm!228

Fix bug in merge_mp_partitions for handling recent checkpoints.

See merge request ADLR/megatron-lm!226

Handle empty documents in preprocess_data.

See merge request ADLR/megatron-lm!225

Init CI tests with very basic import test.

See merge request ADLR/megatron-lm!224

added option to change tensorboard queue size

See merge request ADLR/megatron-lm!223

added options for tensorboard logging

See merge request ADLR/megatron-lm!222

Typo fix.

See merge request ADLR/megatron-lm!221

Teach merge_mp_partitions how to write out a pipelined model.

See merge request ADLR/megatron-lm!218

license text for autoaugmentation

See merge request ADLR/megatron-lm!220

Rework handling of older checkpoint's attention weight/bias ordering.

See merge request ADLR/megatron-lm!219

Move rearranging query_key_value and key_value values in old checkpoints to when the checkpoint is loaded instead of runtime..

added init method std to gpt3 example

See merge request ADLR/megatron-lm!217