GitLab

Add Azure tutorial text and scripts

adding figure caption

* Added features.md and Getting Started guide.

* trimming newlines

* Adding hostfile discussion

* include/exclude emphasis

* Edits overview

* minor edits on the overview

* minor edits

* Fixing broken GPT2 links

* Clarify our relationship with Megatron-LM

* Update README.md

* Update README.md

* minor edits

* fix formatting
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>

* simplifying the batch config, using a single assert to test for validity and allowing for specifying only the micro batch size

* Simplifying Batch Config, Adding ability to specify batch using just micro_batch, and adding a bunch of unit tests

* ran formatting

* Typo fixes and added the config file

* reformatting

* path fixes

* removing print statements

* Ported DeepSpeed overview.

* Renamed subsection

* Formatting table of contents

* initial import of Megatron tutorial

* Grammatical edits, formatting, and paths.

* formatting and data download instructions

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* formatting tutorial

* new perf chart

* removing TODO

* adding pointer to tutorial

* edits

* azure to low bandwidth
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

* add tutorial doc for CIFAR-10 model

* pass pre-commit

* update cifar link to DeepScaleExamples

* simplify the running command

* change deepspeed.pt to deepspeed

Unit tests for add_XXX_arguments

Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Azure pipelines build badge

* Update README.md

* add license badge

* update examples submodule

* install requirements.txt with install script

* add dockerfile

* Enables NCCL backend in @distributed_test

* Adds pytest-forked to avoid CUDA re-initialization issue.

* paste typo

* transcription typo

* Adding testing documentation to README.md

* Add pytest to requirements.txt

update install to use pdcp to distribute wheels

* add allreduce test

* comment out set rank to cuda for now

* switched back to gloo

* Adds distributed_test decorator and some unit tests.

* Setting NCCL backend.

* Parametrizes test.

* rank -> local_rank

* Temporarily disable CUDA initialization.

Fixing file permissions.

Lamb CUDA Kernels

Different Optimizers in DeepSpeed.