Commits · 4a3234e0ab3a6b0039741d7ea2fe0c1b82584763 · OpenDAS / deepspeed

06 Jul, 2020 1 commit

ZeRO-2: Handle gradients of empty partitions (#275) · 4a3234e0

Olatunji Ruwase authored Jul 06, 2020



* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

4a3234e0

29 Jun, 2020 1 commit
- remove tensorflow from reqs (#282) · 97787881
  Jeff Rasley authored Jun 29, 2020
  
  97787881
25 Jun, 2020 1 commit
- bump DSE commit (#278) · 4d64b5ba
  Jeff Rasley authored Jun 25, 2020
  
  4d64b5ba
24 Jun, 2020 1 commit

Improving deepspeed.ai website (#269) · 6379292c

Conglong Li authored Jun 23, 2020

* syntax/typo fix

* add README for documentation

* fix links

* update navigation

* typo fix

* docs readme fix

6379292c

23 Jun, 2020 1 commit

Handle parameter groups smaller than DP (#273) · 88c319aa

Olatunji Ruwase authored Jun 23, 2020

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

88c319aa

20 Jun, 2020 4 commits
- Ai scale (#271) · 664fa30c
  Shaden Smith authored Jun 19, 2020
  
  664fa30c
- Update deepspeed_utils.py (#270) · 224494bd
  Samyam Rajbhandari authored Jun 19, 2020
```
* Removing handle_overflow debugging code in deepspeed_utils.py

* Removing handle_overflow debugging code in deepspeed_zero_optimizer.py

Removing unnecessary overflow handle code. Not sure why it was there in the first place.
```
  224494bd
- Revert "Load non-DeepSpeed checkpoints into ZeRO optimizer" · 02c392f0
  Shaden Smith authored Jun 19, 2020
```
This reverts commit 54c0267e.
```
  02c392f0
- Load non-DeepSpeed checkpoints into ZeRO optimizer · 54c0267e
  Tunji Ruwase authored Jun 20, 2020
  
  54c0267e
17 Jun, 2020 3 commits
- new transformer pre-ln image (#268) · 2e6d93e0
  Shaden Smith authored Jun 17, 2020
  
  2e6d93e0
- add quotes around user argument values (#267) · fedd0386
  Jeff Rasley authored Jun 17, 2020
  
  fedd0386
- adds gtag for website analytics (#266) · 9fcdc885
  Shaden Smith authored Jun 16, 2020
  
  9fcdc885
16 Jun, 2020 1 commit

add the fine-tuning results (#260) · 6e87251c

RezaYazdaniAminabadi authored Jun 16, 2020

* add the fine-tuning results

* updating tutorial and blog-post

* updated the tutorials and links

6e87251c

12 Jun, 2020 1 commit
- minor refactor loss scaler (#261) · 96c4daab
  Chunyang Wen authored Jun 12, 2020
  
  96c4daab
09 Jun, 2020 1 commit

install update: no-sudo + clean build files (#258) · f5025506

Jeff Rasley authored Jun 09, 2020



* install update: no-sudo + clean build files
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

f5025506

08 Jun, 2020 1 commit
- fix transformer kernel preln config (#257) · fc1de4ff
  eltonzheng authored Jun 08, 2020
  
  fc1de4ff
05 Jun, 2020 1 commit

Support migration to FP16 optimizer (#249) · 2312f04b

Olatunji Ruwase authored Jun 05, 2020

* Debugging

* Fix step() bug; Make step timing optional

* Remove unnecessary changes

* Format fixes

* Replace list with scalar variable

* Remove redundant code

* Fix typo

2312f04b

04 Jun, 2020 4 commits
- update DSE commit (#254) · f73d717c
  Jeff Rasley authored Jun 04, 2020
  
  f73d717c
- Specify num_replicas and rank when creating sampler (#216) · 0f72988d
  Vidush Vishwanath authored Jun 04, 2020
  
  0f72988d
- Add log util (#230) · e1ad8803
  Chunyang Wen authored Jun 05, 2020
```
* Add log util

* replace all occurrences of print and logging

* address format

* disable propagate to avoid duplicate log
```
  e1ad8803
- Features links (#252) · 2a1c5db1
  Shaden Smith authored Jun 04, 2020
```
* links and formatting
```
  2a1c5db1
03 Jun, 2020 2 commits
- Update paths; include information about submodule. (#250) · fd2a8fd5
  Ammar Ahmad Awan authored Jun 03, 2020
```
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
```
  fd2a8fd5
- reduce memcpy between host and device (#248) · 8353c594
  eltonzheng authored Jun 03, 2020
  
  8353c594
30 May, 2020 1 commit
- update tests · bbd8cd7d
  Jeff Rasley authored May 29, 2020
  
  bbd8cd7d
29 May, 2020 8 commits

adding overlap_comm documentation (#247) · e04e4016
Samyam Rajbhandari authored May 29, 2020

e04e4016
Point BERT pretraining tutorial to new perf tuning. (#246) · b1ddea7f
Shaden Smith authored May 29, 2020

b1ddea7f
update bert images (#245) · c5c9aaaa
Jeff Rasley authored May 29, 2020

c5c9aaaa
center images (#244) · 0c77f878
Shaden Smith authored May 29, 2020

0c77f878
Image links (#243) · c7d0b0ca
Shaden Smith authored May 29, 2020

c7d0b0ca
update bert tutorial · 6622de16
Jeff Rasley authored May 29, 2020

6622de16
remove old img · dddc526e
Jeff Rasley authored May 29, 2020

dddc526e

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991

28 May, 2020 2 commits

fix: typo (#238) · b652395e
Chunyang Wen authored May 29, 2020
```
* fix: typo in code docs

* more pythonic code
```
b652395e

remove redundant init code (#234) · 6fe0edb8

Chunyang Wen authored May 28, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

6fe0edb8

27 May, 2020 5 commits

if using lamb force legacy_fusion=True (#236) · a2b75527
Jeff Rasley authored May 27, 2020

a2b75527

add support for predivide as a config option (#235) · bc36b91d

Jeff Rasley authored May 27, 2020

* add support for predivide as a flag
* add predivide json config, remove allgather_disable (as it's not currently used anymore)

bc36b91d

Default Contiguous Gradients False (#239) · 01e848b3

Samyam Rajbhandari authored May 27, 2020

Contiguous Gradients should be set to false by default. Its not useful unless the model is very large

01e848b3

Samyamr/cpu memory bloat fix zero (#233) · d24d3de9

Samyam Rajbhandari authored May 27, 2020

* Fix for CPU memory Bloating Issue caused by pyorch backward graph creation in allgather. Fixed by calling detach on tensors before calling all_gather

* Fix for CPU memory Bloating Issue caused by pyorch backward graph creation in allgather. Fixed by calling detach on tensors before calling all_gather

* Fix for CPU memory Bloating Issue caused by pyorch backward graph creation in allgather. Fixed by calling detach on tensors before calling all_gather

d24d3de9

Support fp32 grad clipping and fix max_grad_norm confusion (#232) · abe2204d
Jeff Rasley authored May 26, 2020
```
* updates to support fp32 grad clipping and disable max_grad_norm
```
abe2204d

26 May, 2020 1 commit
- updating website packages (#231) · 6a9d57f6
  Shaden Smith authored May 26, 2020
  
  6a9d57f6