Commits · a76572dc7cc7030b6208cf1dc3e2f6e501d460c8 · OpenDAS / deepspeed

25 Mar, 2020 1 commit
- Adding static loss scaling for ZeRO. (#166) · a76572dc
  Shaden Smith authored Mar 25, 2020
  
  a76572dc
10 Mar, 2020 1 commit

Make lr schedulers support fp16 optimizers (#124) · 1c0b326e

Olatunji Ruwase authored Mar 10, 2020



* add tests cases for onecycle policy with fp16/zero

* Make lr schedulers support fp16 optimizers

* Fix formatting

* More specific naming
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

1c0b326e

26 Feb, 2020 1 commit

Init distributed torch only if needed (#108) · 5aa58b38

Jeff Rasley authored Feb 26, 2020

* add auto-detect to torch dist init

* update tests to infer distributed init status

* prevent crash if dist_init_required is True but already initiliazed

* only init if safe to do so (forgot to add this file in prev commit)

5aa58b38

20 Feb, 2020 1 commit
- Refactor simple model test, fix pythonpath issue (#96) · 001abe23
  Jeff Rasley authored Feb 20, 2020
```
Also a fix for #94 
```
  001abe23
15 Feb, 2020 1 commit
- Fix issue with empty grads for non-fused optimizers (#83) · 807480a0
  Jeff Rasley authored Feb 14, 2020
```
bug fixes for adamw/lamb and corresponding tests
```
  807480a0