Commits · e0f36ed5a1ebc7b1ed044bf459f3576f30ff949a · OpenDAS / deepspeed

11 Mar, 2021 1 commit

Add optimizers and schedules to RTD and updated the corresponding part in the website (#799) · e0f36ed5

Cheng Li authored Mar 11, 2021



* add optimizers and schedules to rtd

* update ds website and fix links

* add optimizers and schedules to rtd

* update ds website and fix links

* add flops profiler to rtd

* fix
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

e0f36ed5

08 Mar, 2021 1 commit

ZeRO 3 Offload (#834) · 599258f9

Samyam Rajbhandari authored Mar 08, 2021



* Squash stage3 v1 (#146)
Co-authored-by: Samyam <samyamr@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

* Fix correctness bug (#147)

* formatting fix (#150)

* stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)

* fp16 Z3 API update and bugfix

* revert debug change

* ZeRO-3 detach and race condition bugfixes (#149)

* trying out ZeRO-3 race condition fix

* CUDA sync instead of stream

* reduction stream sync

* remove commented code

* Fix optimizer state_dict KeyError (#148)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)

* Simplifying the logic for getting averaged gradients (#153)

* skip for now

* Z3 Docs redux (#154)

* removing some TODOs and commented code (#155)

* New Z3 defaults (#156)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* formatting

* megatron external params
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

599258f9

07 Jan, 2021 1 commit
- Add deepspeed.init_distributed to RTD page (#645) · 4e2dc4e4
  Jeff Rasley authored Jan 07, 2021
```
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
```
  4e2dc4e4
15 Dec, 2020 1 commit
- Fixes for RTD build errors (#606) · 6380ee35
  Jeff Rasley authored Dec 15, 2020
```
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
```
  6380ee35
30 Oct, 2020 1 commit

Add CPUAdam optimizer for zero-offload in deepspeed engine (#484) · f5aa2547

Reza Yazdani authored Oct 30, 2020



* add adamW to CPU-ADAM implementation

* supporting cpu-adam optimizer for zero-offload on deepspeed side

* bump DSE to match cpu-adam updates
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

f5aa2547

17 Sep, 2020 1 commit
- readthedocs yaml configuration (#410) · 5812e845
  Shaden Smith authored Sep 16, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  5812e845
10 Sep, 2020 2 commits
- readthedocs upgrade (#402) · c82756cd
  Shaden Smith authored Sep 10, 2020
  
  c82756cd
- Pipeline parallel training engine. (#392) · 65c2f974
  Shaden Smith authored Sep 09, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  65c2f974
29 May, 2020 1 commit

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991

19 May, 2020 1 commit

ZeRO-2 (#217) · f2ac7eaf

Jeff Rasley authored May 19, 2020



Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

f2ac7eaf

22 Apr, 2020 1 commit
- README and RTD improvements. (#198) · dd166ee6
  Shaden Smith authored Apr 21, 2020
  
  dd166ee6
17 Mar, 2020 1 commit
- drafting Jekyll webpage (#143) · 5042dc00
  Shaden Smith authored Mar 17, 2020
  
  5042dc00