GitLab

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

hidden_size % attention_heads == 0 is handled above when dealing with kv_channels.

Adding check for decoder sequence length.

- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions