GitLab

* turn off multi-node launch if only 1 node

* update fan out flag for pdsh

* add fix and tests for get_lr from lr_scheduler before training starts

Add webinar on-demand links and update readme

The parenthesis alter the evaluation of the assert() and it will always evaluate to True.

mpu object is bound to the class instance.. 

the if statement uses  `self.mpu'  but just `mpu` is called in the following lines.. 

This raises a NameError

* fix nv_peer_mem version in dockerfile

* fix security issue, remove pillow dependency (this is only needed for cifar example which has its own requirements.txt)

* updates to amp to support grad clip and grad accumulation
* zero grad using optimizer if in amp mode

* Avoid deadlock for unsynchronized non-zero checkpointing

* Fix formatting issues
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* empty grad fix
* add unit tests for empty grad

* Support saving and loading ZeRO checkpoints on different data
parallelism degree.

* Fix formatting

* Support checkpoint with varying GPU count in ZeRO stage 1

* Fix formatting

* Formatting fixes

* Update model tests

* Remove pprint

* Minor fix

* Fix formatting

* Update model tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* add amp docs

* add amp support for deepspeed (non-ZeRO)
* tests for amp mode

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* syntax/typo fix

* add README for documentation

* fix links

* update navigation

* typo fix

* docs readme fix

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Removing handle_overflow debugging code in deepspeed_utils.py

* Removing handle_overflow debugging code in deepspeed_zero_optimizer.py

Removing unnecessary overflow handle code. Not sure why it was there in the first place.

This reverts commit 54c0267e.

* add the fine-tuning results

* updating tutorial and blog-post

* updated the tutorials and links