GitLab

* BERT title

Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

* remove session_params in deepspeed_constants.py

* add constants info at README.md

* Better config filename

* Clean up configuration ToC

* fix docs permalink

* fix docs permalink

* add support for deepspeed env file to pass custom env values

* simplify deepspeed config example

* add text about mpirun

* Fix broken link for the 1Cycle doc.

* Removed the 1Cycle link from README.md.

* Importing 1Cycle tutorial.

* image paths

* Added LR schedule figure

* line wrap

* lowercase name

* Updating README links

* typo

* Increasing section headers

* Move testing under contributing
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

Add Azure tutorial text and scripts

* Added features.md and Getting Started guide.

* trimming newlines

* Adding hostfile discussion

* include/exclude emphasis

* Edits overview

* minor edits on the overview

* minor edits

* Fixing broken GPT2 links

* Clarify our relationship with Megatron-LM

* Update README.md

* Update README.md

* minor edits

* fix formatting
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>

* Ported DeepSpeed overview.

* Renamed subsection

* Formatting table of contents

* initial import of Megatron tutorial

* Grammatical edits, formatting, and paths.

* formatting and data download instructions

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* new perf chart

* removing TODO

* adding pointer to tutorial

* edits

* azure to low bandwidth
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

* Azure pipelines build badge

* Update README.md

* add license badge

* Enables NCCL backend in @distributed_test

* Adds pytest-forked to avoid CUDA re-initialization issue.

* paste typo

* transcription typo

* Adding testing documentation to README.md

* Add pytest to requirements.txt