GitLab

* Push to remote

* Correctly handle multi output models by doing loss scaling in backward()
Unit tests for multi output models

* Fix formatting issues

* Formatting issues fix

* Fix formatting

* Update DeepSpeedExamples submodule
Enable Megatron model tests

* added zero_allow_untested_optimizer flag helpers

* add zero_allow_untested_optimizer config constants

* zero_allow_untested_optimizer logic with assertion

* Added unit test and CustomOptimizer helper class

* remove session_params in deepspeed_constants.py

* add constants info at README.md

* Better config filename

* Clean up configuration ToC

* fix docs permalink

* fix docs permalink

* Add coming soon to posts

* Add what's new section to main page

GitHub created a CNAME for us automatically. Cool.

* add support for torch 1.3+ builds inside a docker build environment
* remove apex imports

* allow installing a specific apex commit

* Enhancement: Ability to load checkpoint without loading the optimizer states. Unittest testing saving and loading checkpoint with fused, unfused and zero optimizer. The unitest takes about 165s

* add tests cases for onecycle policy with fp16/zero

* Make lr schedulers support fp16 optimizers

* Fix formatting

* More specific naming
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Add --no_sudo to run without sudo

* Add --pip_mirror to set the pip mirror

* Default to running pip without sudo

* Typo

* Add --pip_sudo to Dockerfile and azure-pipelines.yml
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* add support for deepspeed env file to pass custom env values

* simplify deepspeed config example

* add text about mpirun

* add mpirun support for openmpi 4.0

* add master addr support from args

* switch mpi detection to use mpi4py

* set constant for default distributed port

* Make sure deepspeed_mpi exits in args

* add auto-detect to torch dist init

* update tests to infer distributed init status

* prevent crash if dist_init_required is True but already initiliazed

* only init if safe to do so (forgot to add this file in prev commit)

* Update scripts to handle cases where you have other VMs in your sub

* Support subs with other VMs and fix for PDSH permission error

* Minor fix to support subs with other VMs

* Added shutdown with or without delete VM option

In Azure deallocate is like machine shutdown (and prevents billing). You can restart deallocated VM. To fully drop the VM delete is used. This command with "-d" option will fully delete the VM. Without any argument it justs deallocates / shutd down the VM.

* Update scripts to handle cases where you have other VMs in your sub

* Support subs with other VMs and fix for PDSH permission error

* Minor fix to support subs with other VMs

* Removes DeepSpeedDataSource

* dropping unused imports
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* Support legacy optimizer fusion as config option

* Configure for legacy optimizer fusion

* Update configuration jsons for new apex