GitLab

* Adding gradient accumulation support for ZeRO Stage 2. Changing all Megatron-LM tests to also test gradient accumulation

* Gradient Accumulation support for Stage 2. Model tests added to test the feature

* formatting

* Update deepspeed_light.py

removing comment

* Update ds_config_func_bs8_zero1.json

reverting this file back. Its not needed for this PR

* defining baseline prefix
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Update deepspeed_checkpointing.py

* formatting
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Create CODEOWNERS

* turn off multi-node launch if only 1 node

* update fan out flag for pdsh

* add fix and tests for get_lr from lr_scheduler before training starts

Add webinar on-demand links and update readme

The parenthesis alter the evaluation of the assert() and it will always evaluate to True.

mpu object is bound to the class instance.. 

the if statement uses  `self.mpu'  but just `mpu` is called in the following lines.. 

This raises a NameError

* fix nv_peer_mem version in dockerfile

* fix security issue, remove pillow dependency (this is only needed for cifar example which has its own requirements.txt)

* updates to amp to support grad clip and grad accumulation
* zero grad using optimizer if in amp mode

* Avoid deadlock for unsynchronized non-zero checkpointing

* Fix formatting issues
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* empty grad fix
* add unit tests for empty grad

* Support saving and loading ZeRO checkpoints on different data
parallelism degree.

* Fix formatting

* Support checkpoint with varying GPU count in ZeRO stage 1

* Fix formatting

* Formatting fixes

* Update model tests

* Remove pprint

* Minor fix

* Fix formatting

* Update model tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* add amp docs

* add amp support for deepspeed (non-ZeRO)
* tests for amp mode

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* syntax/typo fix

* add README for documentation

* fix links

* update navigation

* typo fix

* docs readme fix

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Removing handle_overflow debugging code in deepspeed_utils.py

* Removing handle_overflow debugging code in deepspeed_zero_optimizer.py

Removing unnecessary overflow handle code. Not sure why it was there in the first place.

This reverts commit 54c0267e.