Commits · e0f36ed5a1ebc7b1ed044bf459f3576f30ff949a · OpenDAS / deepspeed

08 Mar, 2021 1 commit

Samyam Rajbhandari authored Mar 08, 2021



* Squash stage3 v1 (#146)
Co-authored-by: Samyam <samyamr@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

* Fix correctness bug (#147)

* formatting fix (#150)

* stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)

* fp16 Z3 API update and bugfix

* revert debug change

* ZeRO-3 detach and race condition bugfixes (#149)

* trying out ZeRO-3 race condition fix

* CUDA sync instead of stream

* reduction stream sync

* remove commented code

* Fix optimizer state_dict KeyError (#148)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)

* Simplifying the logic for getting averaged gradients (#153)

* skip for now

* Z3 Docs redux (#154)

* removing some TODOs and commented code (#155)

* New Z3 defaults (#156)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* formatting

* megatron external params
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

599258f9

10 Feb, 2021 1 commit
- Clickable screenshots (#746) · 6ee3b296
  Olatunji Ruwase authored Feb 10, 2021
```
* Fix docstring

* Make screenshots clickable for easier viewing
```
  6ee3b296
20 Jan, 2021 1 commit
- [tutorials] typos (#676) · 7b0bee0b
  Stas Bekman authored Jan 20, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  7b0bee0b
05 Jan, 2021 1 commit
- docs: minor spelling tweaks (#623) · 46d2e287
  brett koonce authored Jan 05, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  46d2e287
11 Nov, 2020 1 commit

Update zero.md tutorial (#495) · 0ad4fd88

Samyam Rajbhandari authored Nov 11, 2020



* Update zero.md

Update to ZeRO tutorial to specify the use of activation checkpointing

* Update zero-offload.md

Use activation checkpointing with ZeRO-Offload
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

0ad4fd88

24 Sep, 2020 1 commit

Fix urls in tutorial (#436) · 5d40f006

Conglong Li authored Sep 24, 2020

* url fix

* revert absolute path but keep some actual fix

* add real readme

5d40f006

10 Sep, 2020 1 commit

ZeRO tutorials (#384) · 2dea61f2

Olatunji Ruwase authored Sep 10, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

2dea61f2