Commits · fa87a73a8a3bead24ad9ea52090646fa620d74e8 · OpenDAS / deepspeed

16 Mar, 2021 1 commit
- Fix ZeRO3 save_checkpoint (#857) · fa87a73a
  Olatunji Ruwase authored Mar 16, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  fa87a73a
08 Mar, 2021 1 commit

Samyam Rajbhandari authored Mar 08, 2021



* Squash stage3 v1 (#146)
Co-authored-by: Samyam <samyamr@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

* Fix correctness bug (#147)

* formatting fix (#150)

* stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)

* fp16 Z3 API update and bugfix

* revert debug change

* ZeRO-3 detach and race condition bugfixes (#149)

* trying out ZeRO-3 race condition fix

* CUDA sync instead of stream

* reduction stream sync

* remove commented code

* Fix optimizer state_dict KeyError (#148)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)

* Simplifying the logic for getting averaged gradients (#153)

* skip for now

* Z3 Docs redux (#154)

* removing some TODOs and commented code (#155)

* New Z3 defaults (#156)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* formatting

* megatron external params
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

599258f9

29 Jan, 2021 1 commit
- Dist testing backend fixes, etc. (#708) · 2e2dd861
  Jeff Rasley authored Jan 29, 2021
  
  2e2dd861
14 Jan, 2021 1 commit
- Validate consistent ckpt tags across ranks (#667) · f032e56f
  Jeff Rasley authored Jan 14, 2021
  
  f032e56f
23 Dec, 2020 1 commit
- Elastic training support (#602) · 81aeea36
  Jeff Rasley authored Dec 22, 2020
```
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
```
  81aeea36
02 Dec, 2020 1 commit
- Add 'latest' checkpoint save/load support (#569) · 845921b3
  Jeff Rasley authored Dec 02, 2020
  
  845921b3
21 Nov, 2020 1 commit
- Support non-tensor state in checkpoint (#548) · 6021b702
  Olatunji Ruwase authored Nov 21, 2020
  
  6021b702
18 Nov, 2020 1 commit

Fix layout bug in ZeRO Stage 1 checkpoint logic (#531) · 7752dc5e

Olatunji Ruwase authored Nov 17, 2020

* Fix layout bug in ZeRO Stage 1 checkpoint logic
Add elastic checkpoint option for ZeRO stage 1, default to True

* Format fixes

7752dc5e

12 Nov, 2020 1 commit

DeepSpeed JIT op + PyPI support (#496) · 31f46fee

Jeff Rasley authored Nov 12, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

31f46fee

15 Sep, 2020 1 commit
- pytest skips for tests requiring certain ops (#411) · 91b4a93d
  Jeff Rasley authored Sep 15, 2020
```
* add pytest skips around tests that require certain ops to be installed
```
  91b4a93d
10 Sep, 2020 2 commits

Pipeline parallel training engine. (#392) · 65c2f974
Shaden Smith authored Sep 09, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
65c2f974

ZeRO-Offload release (#391) · 41db1c2f

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

02 Sep, 2020 1 commit

Sparse attn + ops/runtime refactor + v0.3.0 (#343) · e5bbc2e5

Jeff Rasley authored Sep 01, 2020



* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

e5bbc2e5

15 Jul, 2020 1 commit
- Fix bug in fp32 optimizer state loading (#289) · 607814fe
  Olatunji Ruwase authored Jul 15, 2020
  
  607814fe
27 May, 2020 1 commit
- Support fp32 grad clipping and fix max_grad_norm confusion (#232) · abe2204d
  Jeff Rasley authored May 26, 2020
```
* updates to support fp32 grad clipping and disable max_grad_norm
```
  abe2204d
19 May, 2020 1 commit

ZeRO-2 (#217) · f2ac7eaf

Jeff Rasley authored May 19, 2020



Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

f2ac7eaf

06 May, 2020 1 commit
- Fix global_steps checkpoint loading. (#139) · b2c87edf
  Shaden Smith authored May 06, 2020
  
  b2c87edf
10 Mar, 2020 1 commit

Enhancement: Ability to load checkpoint without loading the optimizer… (#128) · 936117b5

Samyam Rajbhandari authored Mar 10, 2020

* Enhancement: Ability to load checkpoint without loading the optimizer states. Unittest testing saving and loading checkpoint with fused, unfused and zero optimizer. The unitest takes about 165s

936117b5