GitLab

Fused softmax checks and additions from Github (#133)

See merge request ADLR/megatron-lm!312

Checkpoint a set number of individual Transformer layers

See merge request ADLR/megatron-lm!301

Memory optimizations for contiguous buffers

See merge request ADLR/megatron-lm!310

consider the case of pipeline-model prallelism

clean up arugments

argument naming cleanup

update readme and examples

bug fix for previous MR (Flag to call empty_cache() each iteration, to reduce fragmentation)

See merge request ADLR/megatron-lm!309

minor changes from github issues

See merge request ADLR/megatron-lm!308

Flag to call empty_cache() each iteration, to reduce fragmentation

See merge request ADLR/megatron-lm!306

simplified the iteration read check across ranks

See merge request ADLR/megatron-lm!307

some small PRs from github

See merge request ADLR/megatron-lm!305

added across ranks sync for checkpoint iteration load and couple of other fixes

See merge request ADLR/megatron-lm!304

Merge branch 'update-sample-bert-pretrain-args' of https://github.com/roclark/Megatron-LM into github_small_prs

added across rank sync for checkpoint iteration laod, fixed type for timing, and validation iterations

changed torch distributed init method from tcp to env

See merge request ADLR/megatron-lm!297

Some tests expect a clean model parallel slate and complain if a previous test left something behind; this change clears more variables that the tests complain about.