Commits · 7833aed20222ea9052499f15c5ed9006c04bacfc · OpenDAS / deepspeed

27 Jan, 2021 1 commit
- Create torch16.yml (#699) · 7833aed2
  Jeff Rasley authored Jan 27, 2021
  
  7833aed2
26 Jan, 2021 1 commit

Fix wrong idx bug in invertible LayerNormBackward1 (#692) · 5221832e

Ying Xiong authored Jan 26, 2021



* fix wrong idx bug in invertible LayerNormBackward1

this index bug cause wrong scale grad

* fix unexpected deletion

* fix idx for LayerNormBackward1_fused_add

* move pos defination in LayerNormBackward1 kernels

* fix format error
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

5221832e

25 Jan, 2021 1 commit
- Add optional timeout parameter to deepspeed.init_distributed (#637) · 852c524a
  sdtblck authored Jan 25, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  852c524a
20 Jan, 2021 3 commits
- Fix ZeRO 2 + Pipelining (#677) · 34c83a5a
  Leo Gao authored Jan 20, 2021
```
* Fix ZeRO 2 + Pipelining
```
  34c83a5a
- make test_pipe more stable (#683) · e59ba12d
  Shaden Smith authored Jan 20, 2021
  
  e59ba12d
- [tutorials] typos (#676) · 7b0bee0b
  Stas Bekman authored Jan 20, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  7b0bee0b
19 Jan, 2021 1 commit
- add zero-offload paper (#680) · 82cecf69
  Jeff Rasley authored Jan 19, 2021
```
* Update README.md

* Update index.md
```
  82cecf69
15 Jan, 2021 5 commits
- doc fix (#651) · 7b07e123
  Stas Bekman authored Jan 15, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  7b07e123
- add missing config menu entries (#652) · e729a3fd
  Stas Bekman authored Jan 15, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  e729a3fd
- Add AdamW to the supported optimizers (#672) · c5e42641
  Stas Bekman authored Jan 15, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  c5e42641
- skip empty lines in hostfile (#669) · 6217a6c2
  Jeff Rasley authored Jan 15, 2021
  
  6217a6c2
- Support optimizer AdamW type (#670) · 865104be
  Olatunji Ruwase authored Jan 15, 2021
  
  865104be
14 Jan, 2021 1 commit
- Validate consistent ckpt tags across ranks (#667) · f032e56f
  Jeff Rasley authored Jan 14, 2021
  
  f032e56f
13 Jan, 2021 2 commits

Move workspace memory-allocation to PyTorch (#661) · 981bc7d4

Reza Yazdani authored Jan 12, 2021

* move workspace memory-allocation to PyTorch

* refine the code based on the comments

* remove unnecessary options

* remove bsz from set_seq_len function

981bc7d4

squash latest flops profiling changes (#1) (#664) · e2fbe4d2

Cheng Li authored Jan 12, 2021


Co-authored-by: Cheng Li <pistasable@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

e2fbe4d2

12 Jan, 2021 1 commit
- Handle actvitation checkpointing args that are None or non-tensors (#660) · adcfd269
  Shaden Smith authored Jan 12, 2021
```
Special thanks to @g-karthik for tracking this issue down.
```
  adcfd269
08 Jan, 2021 5 commits

LR scheduler unit tests (#429) · da5563a9

Olatunji Ruwase authored Jan 08, 2021



* Add Linear warmup+decay lr schedule
Update lr schedule unit tests

* LR scheduler unit tests for LR Range Test and 1Cycle

* Disable yapf to preserve parameterizaton

* Disable test_pipe.py for CI debugging

* Disable test_lr_scheduler for CI debugging

* Disable test_lr_scheduler for CI debugging

* Enable all unit tests for CI debugging
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

da5563a9

version bump to 0.3.10 · c14b839d
Jeff Rasley authored Jan 08, 2021

c14b839d
Remove a very verbose print statement. (#649) · af212f66
Ammar Ahmad Awan authored Jan 08, 2021
```
* Remove a very verbose print statement.

* Update engine.py
```
af212f66
add additional validation checks in elastic config (#646) · bc046dc4
Jeff Rasley authored Jan 08, 2021

bc046dc4
document deepspeed.initialize() (#644) · 828d75ba
Stas Bekman authored Jan 08, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
828d75ba

07 Jan, 2021 3 commits

Add deepspeed.init_distributed to RTD page (#645) · 4e2dc4e4
Jeff Rasley authored Jan 07, 2021
```
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
```
4e2dc4e4

Bump nokogiri from 1.10.10 to 1.11.0 in /docs (#630) · 8cea96dd

dependabot[bot] authored Jan 07, 2021

Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.10 to 1.11.0.
- [Release notes](https://github.com/sparklemotion/nokogiri/releases)
- [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.10.10...v1.11.0

)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

8cea96dd

Update builder.py (#642) · 64461da4
Xingjian Shi authored Jan 06, 2021

64461da4

06 Jan, 2021 1 commit

Module replacement support (#586) · 44bd538b

Jeff Rasley authored Jan 06, 2021


Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>

44bd538b

05 Jan, 2021 4 commits
- Fix docstring format (#640) · 5ab12795
  Olatunji Ruwase authored Jan 05, 2021
  
  5ab12795
- docs: minor spelling tweaks (#623) · 46d2e287
  brett koonce authored Jan 05, 2021
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  46d2e287
- change dist to torch.distributed to fix bug in assert. (#638) · d38ad6a1
  Ammar Ahmad Awan authored Jan 05, 2021
  
  d38ad6a1
- Allow DeepSpeed models to be initialized with optimizer=None (#469) · a9a83a6f
  gcooper-isi authored Jan 05, 2021
```
Allow DeepSpeed models to be initialized with optimizer=None
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
```
  a9a83a6f
04 Jan, 2021 2 commits
- Support initialization with dict configuration (#632) · e6ac7311
  Olatunji Ruwase authored Jan 04, 2021
  
  e6ac7311
- update SA comp check to fix torch-cpu issue (#631) · 24e07399
  Jeff Rasley authored Jan 04, 2021
  
  24e07399
23 Dec, 2020 1 commit
- Elastic training support (#602) · 81aeea36
  Jeff Rasley authored Dec 22, 2020
```
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
```
  81aeea36
18 Dec, 2020 1 commit
- Ability to initialize distributed backend outside deepspeed runtime (#608) · 7435b2f1
  Jeff Rasley authored Dec 17, 2020
  
  7435b2f1
17 Dec, 2020 1 commit
- Transformer-kernel - supporting any arbitrary sequence-length (#587) · fd2f970b
  Reza Yazdani authored Dec 17, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  fd2f970b
15 Dec, 2020 2 commits

Fixes for RTD build errors (#606) · 6380ee35
Jeff Rasley authored Dec 15, 2020
```
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
```
6380ee35

[doc] xref to hostfile discussion (#604) · 007466e5

Stas Bekman authored Dec 15, 2020

* [doc] xref to hostfile discussion

wasn't clear where to find what was meant by `hostfile` - so adding a link to where it's discussed.

* remove whitespace

007466e5

14 Dec, 2020 1 commit
- implement missing get_last_lr (#595) · 9f8e8f38
  Stas Bekman authored Dec 14, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  9f8e8f38
11 Dec, 2020 3 commits
- Update launcher to set local rank environ variable (#597) · c5a449f9
  Jeff Rasley authored Dec 11, 2020
```
* Update launch.py

* formatting
```
  c5a449f9
- Supported customizing kwargs for lr_scheduler (#584) · a4763f55
  carefree0910 authored Dec 12, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  a4763f55
- add DeepSpeedZeroConfig repr method (#596) · 66268bd3
  Stas Bekman authored Dec 11, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  66268bd3