GitLab

* adding BingSqaud e2e test

* updating the draft test; bring final step under try section

* finalizinf test for base deepspeed and deepspeed with ZeRO

* applying the comment (thanks Jeff); fixed formatting

* update Sparse Attention Tutorial

* fixed few issues and applied comments for better organization and readability

* updated sparse attention tutorial with making how to use section incremental; applying more comments
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>

* adding link to Sparse Attention in Navigation page

Remove llvm/cmake install for now, causing pyyaml issues

* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

Renaming config files to gas3

* Adding gradient accumulation support for ZeRO Stage 2. Changing all Megatron-LM tests to also test gradient accumulation

* Gradient Accumulation support for Stage 2. Model tests added to test the feature

* formatting

* Update deepspeed_light.py

removing comment

* Update ds_config_func_bs8_zero1.json

reverting this file back. Its not needed for this PR

* defining baseline prefix
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Update deepspeed_checkpointing.py

* formatting
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Create CODEOWNERS

* turn off multi-node launch if only 1 node

* update fan out flag for pdsh

* add fix and tests for get_lr from lr_scheduler before training starts

Add webinar on-demand links and update readme

The parenthesis alter the evaluation of the assert() and it will always evaluate to True.

mpu object is bound to the class instance.. 

the if statement uses  `self.mpu'  but just `mpu` is called in the following lines.. 

This raises a NameError

* fix nv_peer_mem version in dockerfile

* fix security issue, remove pillow dependency (this is only needed for cifar example which has its own requirements.txt)

* updates to amp to support grad clip and grad accumulation
* zero grad using optimizer if in amp mode