GitLab

The paper has this info, so proposing to copy it next to the table. 

Otherwise it's hard to guess whether you used 40GB A100s or 80GB ones (and secondary, n_gpus per node).

Thank you!

Remove duplicated bulletpoint

PP seems to have been added in https://github.com/NVIDIA/Megatron-LM/commit/46c74b4ca06a7794db1e2615544095535cdf12c2, so I think this clause is not accurate anymore

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>

Signed-off-by: Evelina Bakhturina <ebakhturina@nvidia.com>