Commits · 1f972427fc6c9b22da97ca8b89c194bfb3988d6f · OpenDAS / deepspeed

21 Jul, 2020 1 commit
- only global rank 0 can log tensorboard data; avoid multi gpu/node race for the log directory (#296) · 1f972427
  Arash Ashari authored Jul 21, 2020
  
  1f972427
15 Jul, 2020 2 commits
- Empty grad fix (#291) · 376818ef
  Jeff Rasley authored Jul 15, 2020
```
* empty grad fix
* add unit tests for empty grad
```
  376818ef
- Fix bug in fp32 optimizer state loading (#289) · 607814fe
  Olatunji Ruwase authored Jul 15, 2020
  
  607814fe
14 Jul, 2020 1 commit

Support loading and saving ZeRO checkpoints with changing DP degree (#240) · 7ccc9daf

Olatunji Ruwase authored Jul 14, 2020



* Support saving and loading ZeRO checkpoints on different data
parallelism degree.

* Fix formatting

* Support checkpoint with varying GPU count in ZeRO stage 1

* Fix formatting

* Formatting fixes

* Update model tests

* Remove pprint

* Minor fix

* Fix formatting

* Update model tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

7ccc9daf

13 Jul, 2020 1 commit
- Update amp docs (#287) · 366d8816
  Jeff Rasley authored Jul 13, 2020
```
* add amp docs
```
  366d8816
11 Jul, 2020 1 commit
- Support amp deepspeed backend (#286) · f5453124
  Jeff Rasley authored Jul 11, 2020
```
* add amp support for deepspeed (non-ZeRO)
* tests for amp mode
```
  f5453124
06 Jul, 2020 1 commit

ZeRO-2: Handle gradients of empty partitions (#275) · 4a3234e0

Olatunji Ruwase authored Jul 06, 2020



* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

4a3234e0

29 Jun, 2020 1 commit
- remove tensorflow from reqs (#282) · 97787881
  Jeff Rasley authored Jun 29, 2020
  
  97787881
25 Jun, 2020 1 commit
- bump DSE commit (#278) · 4d64b5ba
  Jeff Rasley authored Jun 25, 2020
  
  4d64b5ba
24 Jun, 2020 1 commit

Improving deepspeed.ai website (#269) · 6379292c

Conglong Li authored Jun 23, 2020

* syntax/typo fix

* add README for documentation

* fix links

* update navigation

* typo fix

* docs readme fix

6379292c

23 Jun, 2020 1 commit

Handle parameter groups smaller than DP (#273) · 88c319aa

Olatunji Ruwase authored Jun 23, 2020

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

88c319aa

20 Jun, 2020 4 commits
- Ai scale (#271) · 664fa30c
  Shaden Smith authored Jun 19, 2020
  
  664fa30c
- Update deepspeed_utils.py (#270) · 224494bd
  Samyam Rajbhandari authored Jun 19, 2020
```
* Removing handle_overflow debugging code in deepspeed_utils.py

* Removing handle_overflow debugging code in deepspeed_zero_optimizer.py

Removing unnecessary overflow handle code. Not sure why it was there in the first place.
```
  224494bd
- Revert "Load non-DeepSpeed checkpoints into ZeRO optimizer" · 02c392f0
  Shaden Smith authored Jun 19, 2020
```
This reverts commit 54c0267e.
```
  02c392f0
- Load non-DeepSpeed checkpoints into ZeRO optimizer · 54c0267e
  Tunji Ruwase authored Jun 20, 2020
  
  54c0267e
17 Jun, 2020 3 commits
- new transformer pre-ln image (#268) · 2e6d93e0
  Shaden Smith authored Jun 17, 2020
  
  2e6d93e0
- add quotes around user argument values (#267) · fedd0386
  Jeff Rasley authored Jun 17, 2020
  
  fedd0386
- adds gtag for website analytics (#266) · 9fcdc885
  Shaden Smith authored Jun 16, 2020
  
  9fcdc885
16 Jun, 2020 1 commit

add the fine-tuning results (#260) · 6e87251c

RezaYazdaniAminabadi authored Jun 16, 2020

* add the fine-tuning results

* updating tutorial and blog-post

* updated the tutorials and links

6e87251c

12 Jun, 2020 1 commit
- minor refactor loss scaler (#261) · 96c4daab
  Chunyang Wen authored Jun 12, 2020
  
  96c4daab
09 Jun, 2020 1 commit

install update: no-sudo + clean build files (#258) · f5025506

Jeff Rasley authored Jun 09, 2020



* install update: no-sudo + clean build files
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

f5025506

08 Jun, 2020 1 commit
- fix transformer kernel preln config (#257) · fc1de4ff
  eltonzheng authored Jun 08, 2020
  
  fc1de4ff
05 Jun, 2020 1 commit

Support migration to FP16 optimizer (#249) · 2312f04b

Olatunji Ruwase authored Jun 05, 2020

* Debugging

* Fix step() bug; Make step timing optional

* Remove unnecessary changes

* Format fixes

* Replace list with scalar variable

* Remove redundant code

* Fix typo

2312f04b

04 Jun, 2020 4 commits
- update DSE commit (#254) · f73d717c
  Jeff Rasley authored Jun 04, 2020
  
  f73d717c
- Specify num_replicas and rank when creating sampler (#216) · 0f72988d
  Vidush Vishwanath authored Jun 04, 2020
  
  0f72988d
- Add log util (#230) · e1ad8803
  Chunyang Wen authored Jun 05, 2020
```
* Add log util

* replace all occurrences of print and logging

* address format

* disable propagate to avoid duplicate log
```
  e1ad8803
- Features links (#252) · 2a1c5db1
  Shaden Smith authored Jun 04, 2020
```
* links and formatting
```
  2a1c5db1
03 Jun, 2020 2 commits
- Update paths; include information about submodule. (#250) · fd2a8fd5
  Ammar Ahmad Awan authored Jun 03, 2020
```
Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com>
```
  fd2a8fd5
- reduce memcpy between host and device (#248) · 8353c594
  eltonzheng authored Jun 03, 2020
  
  8353c594
30 May, 2020 1 commit
- update tests · bbd8cd7d
  Jeff Rasley authored May 29, 2020
  
  bbd8cd7d
29 May, 2020 8 commits

adding overlap_comm documentation (#247) · e04e4016
Samyam Rajbhandari authored May 29, 2020

e04e4016
Point BERT pretraining tutorial to new perf tuning. (#246) · b1ddea7f
Shaden Smith authored May 29, 2020

b1ddea7f
update bert images (#245) · c5c9aaaa
Jeff Rasley authored May 29, 2020

c5c9aaaa
center images (#244) · 0c77f878
Shaden Smith authored May 29, 2020

0c77f878
Image links (#243) · c7d0b0ca
Shaden Smith authored May 29, 2020

c7d0b0ca
update bert tutorial · 6622de16
Jeff Rasley authored May 29, 2020

6622de16
remove old img · dddc526e
Jeff Rasley authored May 29, 2020

dddc526e

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991

28 May, 2020 2 commits

fix: typo (#238) · b652395e
Chunyang Wen authored May 29, 2020
```
* fix: typo in code docs

* more pythonic code
```
b652395e

remove redundant init code (#234) · 6fe0edb8

Chunyang Wen authored May 28, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

6fe0edb8