Commits · 599258f97992e5a47db9408e4e3622805ce1adb5 · OpenDAS / deepspeed

08 Mar, 2021 1 commit

Samyam Rajbhandari authored Mar 08, 2021



* Squash stage3 v1 (#146)
Co-authored-by: Samyam <samyamr@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

* Fix correctness bug (#147)

* formatting fix (#150)

* stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)

* fp16 Z3 API update and bugfix

* revert debug change

* ZeRO-3 detach and race condition bugfixes (#149)

* trying out ZeRO-3 race condition fix

* CUDA sync instead of stream

* reduction stream sync

* remove commented code

* Fix optimizer state_dict KeyError (#148)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)

* Simplifying the logic for getting averaged gradients (#153)

* skip for now

* Z3 Docs redux (#154)

* removing some TODOs and commented code (#155)

* New Z3 defaults (#156)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* formatting

* megatron external params
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

599258f9

29 Jan, 2021 1 commit
- Dist testing backend fixes, etc. (#708) · 2e2dd861
  Jeff Rasley authored Jan 29, 2021
  
  2e2dd861
15 Jan, 2021 1 commit
- Support optimizer AdamW type (#670) · 865104be
  Olatunji Ruwase authored Jan 15, 2021
  
  865104be
04 Jan, 2021 1 commit
- Support initialization with dict configuration (#632) · e6ac7311
  Olatunji Ruwase authored Jan 04, 2021
  
  e6ac7311
12 Nov, 2020 1 commit

DeepSpeed JIT op + PyPI support (#496) · 31f46fee

Jeff Rasley authored Nov 12, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

31f46fee

15 Sep, 2020 1 commit
- pytest skips for tests requiring certain ops (#411) · 91b4a93d
  Jeff Rasley authored Sep 15, 2020
```
* add pytest skips around tests that require certain ops to be installed
```
  91b4a93d
10 Sep, 2020 2 commits

fix for 16GB v100 nodes (#393) · b1d4bd73
Jeff Rasley authored Sep 10, 2020

b1d4bd73

ZeRO-Offload release (#391) · 41db1c2f

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

15 Jul, 2020 1 commit
- Empty grad fix (#291) · 376818ef
  Jeff Rasley authored Jul 15, 2020
```
* empty grad fix
* add unit tests for empty grad
```
  376818ef
11 Jul, 2020 1 commit
- Support amp deepspeed backend (#286) · f5453124
  Jeff Rasley authored Jul 11, 2020
```
* add amp support for deepspeed (non-ZeRO)
* tests for amp mode
```
  f5453124
06 Jul, 2020 1 commit

ZeRO-2: Handle gradients of empty partitions (#275) · 4a3234e0

Olatunji Ruwase authored Jul 06, 2020



* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

4a3234e0

23 Jun, 2020 1 commit

Handle parameter groups smaller than DP (#273) · 88c319aa

Olatunji Ruwase authored Jun 23, 2020

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

88c319aa

27 May, 2020 1 commit
- Support fp32 grad clipping and fix max_grad_norm confusion (#232) · abe2204d
  Jeff Rasley authored May 26, 2020
```
* updates to support fp32 grad clipping and disable max_grad_norm
```
  abe2204d
19 May, 2020 1 commit

ZeRO-2 (#217) · f2ac7eaf

Jeff Rasley authored May 19, 2020



Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

f2ac7eaf

24 Apr, 2020 1 commit
- Fix index out of range error when parameter count is not multiple of ranks (#202) · 512a0d4d
  Olatunji Ruwase authored Apr 24, 2020
  
  512a0d4d
27 Mar, 2020 1 commit

Add "zero_allow_untested_optimizer" option in conf file (#173) · 43f27332

Calogero Zarbo authored Mar 27, 2020

* added zero_allow_untested_optimizer flag helpers

* add zero_allow_untested_optimizer config constants

* zero_allow_untested_optimizer logic with assertion

* Added unit test and CustomOptimizer helper class

43f27332

25 Mar, 2020 1 commit
- Adding static loss scaling for ZeRO. (#166) · a76572dc
  Shaden Smith authored Mar 25, 2020
  
  a76572dc
10 Mar, 2020 1 commit

Make lr schedulers support fp16 optimizers (#124) · 1c0b326e

Olatunji Ruwase authored Mar 10, 2020



* add tests cases for onecycle policy with fp16/zero

* Make lr schedulers support fp16 optimizers

* Fix formatting

* More specific naming
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

1c0b326e

26 Feb, 2020 1 commit

Init distributed torch only if needed (#108) · 5aa58b38

Jeff Rasley authored Feb 26, 2020

* add auto-detect to torch dist init

* update tests to infer distributed init status

* prevent crash if dist_init_required is True but already initiliazed

* only init if safe to do so (forgot to add this file in prev commit)

5aa58b38

20 Feb, 2020 1 commit
- Refactor simple model test, fix pythonpath issue (#96) · 001abe23
  Jeff Rasley authored Feb 20, 2020
```
Also a fix for #94 
```
  001abe23
15 Feb, 2020 1 commit
- Fix issue with empty grads for non-fused optimizers (#83) · 807480a0
  Jeff Rasley authored Feb 14, 2020
```
bug fixes for adamw/lamb and corresponding tests
```
  807480a0