GitLab

* Activation checkpoint support for non tensor input/output

* Format fixes

* Address PR comments; Add ordering edge case tests

* add -e/--examples flag to checkout submodules

* bump DSE commit

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Use log dist function instead of print

* Expose ranks
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* 1-bit adam doc fix

* 1-bit adam doc fix

* 1-bit adam doc fix
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* work on flops profiler tutorial

* update flops profiler tutorial

* add flops profiler tutorial and fix names

* work on flops profiler tutorial

* update flops profiler tutorial

* add flops profiler tutorial and fix names

* fix tailing ws

* fix names

* remove multistep profiling and update docs

* fix cases where functionals and submodules coexist in a parent module, update readme

* fix typo

* always invoke post hook function

* fix module flops sum and update tests

* update tutorial

* Fix docstring

* Make screenshots clickable for easier viewing

* Improve starred expressions

`deepspeed/profiling/flops_profiler/profiler.py` uses starred expressions
that are no longer valid with [PEP 617][1]. The new Python parser is in 3.9,
and this change allows DeepSpeed to run with the newest Python version. I have
not checked all locations that has this issue. However, this change allows me
to run simple examples.

[1]: https://www.python.org/dev/peps/pep-0617/

* Match style for "Improve starred expressions", although readability suffers

The style guide might need to be updated for this new use case of expressions.
Python [Issue 40631][1] includes more discussion on the change.

[1]: https://bugs.python.org/issue40631

Co-authored-by: Cheng Li <pistasable@gmail.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Add executable permission to `ds_elastic` and `ds_report` in `bin`.

* Automatic `ds_elastic` formatting
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix wrong idx bug in invertible LayerNormBackward1

this index bug cause wrong scale grad

* fix unexpected deletion

* fix idx for LayerNormBackward1_fused_add

* move pos defination in LayerNormBackward1 kernels

* fix format error
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Fix ZeRO 2 + Pipelining

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Update README.md

* Update index.md

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* move workspace memory-allocation to PyTorch

* refine the code based on the comments

* remove unnecessary options

* remove bsz from set_seq_len function

Co-authored-by: Cheng Li <pistasable@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Special thanks to @g-karthik for tracking this issue down.

* Add Linear warmup+decay lr schedule
Update lr schedule unit tests

* LR scheduler unit tests for LR Range Test and 1Cycle

* Disable yapf to preserve parameterizaton

* Disable test_pipe.py for CI debugging

* Disable test_lr_scheduler for CI debugging

* Disable test_lr_scheduler for CI debugging

* Enable all unit tests for CI debugging
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Remove a very verbose print statement.

* Update engine.py