GitLab

merge sequence parallelism's layernorm all-reduce into distributed optimizer.

fix for sequence parallelism in bert pooling

See merge request ADLR/megatron-lm!418

Update readme with sequence parallelism and selective activation recompute.

See merge request ADLR/megatron-lm!417

Sequence parallelism + attention checkpoint

See merge request ADLR/megatron-lm!413

Data preprocessing testing changes + fixes

See merge request ADLR/megatron-lm!416

Fourth phase vision merge: classification and segmentation tasks

See merge request ADLR/megatron-lm!400

tools/merge_datasets.py
- tool to merge multiple dataset files into a single dataset
- testing conducted and included in the megatron-testing repo https://gitlab-master.nvidia.com/ADLR/megatron-testing

tools/preprocess_data.py
- magic numbers changed to required command line arguments

megatron/data/indexed_dataset.py
- when merging, fix to properly update document index
- testing conducted and included in the megatron-testing repo (see above)
- fix follows this history https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/66

jit functions warmpups to (1) match fprop and recompute results and (2) remove grad_enable

See merge request ADLR/megatron-lm!404

change dummy_handler to nullcontext