GitLab

* updates to amp to support grad clip and grad accumulation
* zero grad using optimizer if in amp mode

* Avoid deadlock for unsynchronized non-zero checkpointing

* Fix formatting issues
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* empty grad fix
* add unit tests for empty grad

* Support saving and loading ZeRO checkpoints on different data
parallelism degree.

* Fix formatting

* Support checkpoint with varying GPU count in ZeRO stage 1

* Fix formatting

* Formatting fixes

* Update model tests

* Remove pprint

* Minor fix

* Fix formatting

* Update model tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* add amp docs

* add amp support for deepspeed (non-ZeRO)
* tests for amp mode

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* syntax/typo fix

* add README for documentation

* fix links

* update navigation

* typo fix

* docs readme fix

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Removing handle_overflow debugging code in deepspeed_utils.py

* Removing handle_overflow debugging code in deepspeed_zero_optimizer.py

Removing unnecessary overflow handle code. Not sure why it was there in the first place.

This reverts commit 54c0267e.

* add the fine-tuning results

* updating tutorial and blog-post

* updated the tutorials and links

* install update: no-sudo + clean build files
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

* Debugging

* Fix step() bug; Make step timing optional

* Remove unnecessary changes

* Format fixes

* Replace list with scalar variable

* Remove redundant code

* Fix typo

* Add log util

* replace all occurrences of print and logging

* address format

* disable propagate to avoid duplicate log

* links and formatting

Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>