Commits · 473ff985fb8770fbbd9073554a666aac683d57d2 · OpenDAS / deepspeed

10 Sep, 2020 2 commits

Pipeline parallel training engine. (#392) · 65c2f974
Shaden Smith authored Sep 09, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
65c2f974

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

02 Sep, 2020 1 commit

Sparse attn + ops/runtime refactor + v0.3.0 (#343) · e5bbc2e5

Jeff Rasley authored Sep 01, 2020



* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

e5bbc2e5

15 Jul, 2020 1 commit
- Fix bug in fp32 optimizer state loading (#289) · 607814fe
  Olatunji Ruwase authored Jul 15, 2020
  
  607814fe
27 May, 2020 1 commit
- Support fp32 grad clipping and fix max_grad_norm confusion (#232) · abe2204d
  Jeff Rasley authored May 26, 2020
```
* updates to support fp32 grad clipping and disable max_grad_norm
```
  abe2204d
19 May, 2020 1 commit

ZeRO-2 (#217) · f2ac7eaf

Jeff Rasley authored May 19, 2020



Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

f2ac7eaf

06 May, 2020 1 commit
- Fix global_steps checkpoint loading. (#139) · b2c87edf
  Shaden Smith authored May 06, 2020
  
  b2c87edf
10 Mar, 2020 1 commit

Enhancement: Ability to load checkpoint without loading the optimizer… (#128) · 936117b5

Samyam Rajbhandari authored Mar 10, 2020

* Enhancement: Ability to load checkpoint without loading the optimizer states. Unittest testing saving and loading checkpoint with fused, unfused and zero optimizer. The unitest takes about 165s

936117b5