Commits · 599258f97992e5a47db9408e4e3622805ce1adb5 · OpenDAS / deepspeed

08 Mar, 2021 1 commit

Samyam Rajbhandari authored Mar 08, 2021



* Squash stage3 v1 (#146)
Co-authored-by: Samyam <samyamr@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

* Fix correctness bug (#147)

* formatting fix (#150)

* stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)

* fp16 Z3 API update and bugfix

* revert debug change

* ZeRO-3 detach and race condition bugfixes (#149)

* trying out ZeRO-3 race condition fix

* CUDA sync instead of stream

* reduction stream sync

* remove commented code

* Fix optimizer state_dict KeyError (#148)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)

* Simplifying the logic for getting averaged gradients (#153)

* skip for now

* Z3 Docs redux (#154)

* removing some TODOs and commented code (#155)

* New Z3 defaults (#156)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

* formatting

* megatron external params
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: eltonzheng <eltonz@microsoft.com>

599258f9

26 Feb, 2021 1 commit
- fixing the compiling issue for the AMD architecture (#796) · 490e6f7c
  Reza Yazdani authored Feb 26, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  490e6f7c
07 Jan, 2021 1 commit
- Update builder.py (#642) · 64461da4
  Xingjian Shi authored Jan 06, 2021
  
  64461da4
04 Jan, 2021 1 commit
- update SA comp check to fix torch-cpu issue (#631) · 24e07399
  Jeff Rasley authored Jan 04, 2021
  
  24e07399
23 Dec, 2020 1 commit
- Elastic training support (#602) · 81aeea36
  Jeff Rasley authored Dec 22, 2020
```
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
```
  81aeea36
11 Dec, 2020 1 commit
- [build] fix computer capability arch flags, add PTX, handle PTX (#591) · 8a184b6b
  Stas Bekman authored Dec 11, 2020
```
* fix arch flags, add PTX

* bug fix
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  8a184b6b
07 Dec, 2020 2 commits

[build] add compute_86 (#577) · e8b126d9

Stas Bekman authored Dec 07, 2020

RTX-30 series are compute_86
```
python -c "import torch; print(torch.cuda.get_device_capability())"
```
This PR adds support for this compute capability.

Reference: https://developer.nvidia.com/cuda-gpus

Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

e8b126d9

[build] make builder smarter and configurable wrt compute capabilities + docs (#578) · ce363d0e
Stas Bekman authored Dec 07, 2020

ce363d0e

03 Dec, 2020 2 commits
- [build] build against installed cuda-11.1 while torch built w/ cuda-11.0 (#570) · ff58fa7e
  Stas Bekman authored Dec 02, 2020
  
  ff58fa7e
- Add compute capability 8.0 if on cuda 11+ (#572) · be33bea4
  Jeff Rasley authored Dec 02, 2020
  
  be33bea4
14 Nov, 2020 1 commit
- Dependency pruning (#528) · 0dc84200
  Jeff Rasley authored Nov 13, 2020
```
* remove cpu-feature

* remove psutils requirement
```
  0dc84200
12 Nov, 2020 1 commit

DeepSpeed JIT op + PyPI support (#496) · 31f46fee

Jeff Rasley authored Nov 12, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

31f46fee