Commits · a148bd33d6cabdc363754e622a9aa24493ef7512 · OpenDAS / deepspeed

21 Sep, 2020 1 commit
- Add configurable intermediate size to transformer kernels (#423) · a148bd33
  RezaYazdaniAminabadi authored Sep 21, 2020
  
  a148bd33
18 Sep, 2020 3 commits
- Fix activation checkpoint unit tests for GPU systems (#421) · a825f996
  Shaden Smith authored Sep 18, 2020
  
  a825f996
- Revert "Activation checkpointing bugfix and unit tests (#420)" (#422) · a74a604a
  Jeff Rasley authored Sep 18, 2020
```
This reverts commit 01b6e27e

.
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
```
  a74a604a
- Activation checkpointing bugfix and unit tests (#420) · 01b6e27e
  Shaden Smith authored Sep 18, 2020
```
* Activation checkpointing bugfix and unit tests.

* Activation checkpointing bugfix and unit tests.
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  01b6e27e
17 Sep, 2020 3 commits
- Remove pip --use-feature (#419) · 5bc7d4e1
  Jeff Rasley authored Sep 17, 2020
  
  5bc7d4e1
- Fix few typos in the docs (#418) · c66f3881
  Gowtham Prudhvi authored Sep 17, 2020
  
  c66f3881
- readthedocs yaml configuration (#410) · 5812e845
  Shaden Smith authored Sep 16, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  5812e845
16 Sep, 2020 4 commits

Fix a typo in comments (#415) · 4fef478f
Haibin Lin authored Sep 16, 2020
```
* Update stage2.py
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
4fef478f
Overflow fix (#416) · f5cce75e
Shaden Smith authored Sep 16, 2020
```
* Switches fused_optimizer overflow calculation
```
f5cce75e

Olatunji Ruwase authored Sep 16, 2020



* Update installation instructions

* Format fix

* ZeRO tutorial

* Format fixes

* ZeRO-Offload

* ZeRO and ZeRO-Offload tutorials

* Update navigation page

* Format fixes

* Add yuxhe feedback

* Fix blog post link

* Fix OneBit-Adam link
Tweak scheduler example

* Fix date link
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

7d91be97

Add Linear warmup+decay lr schedule (#414) · 0e942df0
Olatunji Ruwase authored Sep 16, 2020
```
Update lr schedule unit tests
```
0e942df0

15 Sep, 2020 3 commits
- add cpu-adam, reformat, add colors (#413) · a9e8325f
  Jeff Rasley authored Sep 15, 2020
  
  a9e8325f
- fix bug related to stitching reduced grads across communication partitions (#318) · 55ed1057
  Jeff Rasley authored Sep 15, 2020
  
  55ed1057
- pytest skips for tests requiring certain ops (#411) · 91b4a93d
  Jeff Rasley authored Sep 15, 2020
```
* add pytest skips around tests that require certain ops to be installed
```
  91b4a93d
13 Sep, 2020 1 commit
- scales throughput by logging freq (#408) · 473ff985
  Shaden Smith authored Sep 13, 2020
  
  473ff985
11 Sep, 2020 2 commits
- Revert "supporting different intermediate sizes other than 4 * hidden_dim (#389)" (#404) · 4ac9bf60
  Jeff Rasley authored Sep 11, 2020
```
This reverts commit e549be60.
```
  4ac9bf60
- supporting different intermediate sizes other than 4 * hidden_dim (#389) · e549be60
  RezaYazdaniAminabadi authored Sep 11, 2020
```
* supporting different intermediate sizes other than 4*hidden_dim

* run precommit

* uncommnet the unit tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  e549be60
10 Sep, 2020 22 commits

readthedocs upgrade (#402) · c82756cd
Shaden Smith authored Sep 10, 2020

c82756cd
remove old pt file · 15ca99c6
Jeff Rasley authored Sep 10, 2020

15ca99c6

Update ZeRO-Offload blog post link (#401) · d15015e9

Olatunji Ruwase authored Sep 10, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

d15015e9

Update news site with press release link · 5dc4d6c7
Jeff Rasley authored Sep 10, 2020

5dc4d6c7
Update _config.yml · ea92ed29
Jeff Rasley authored Sep 10, 2020

ea92ed29
Update README.md · 96935958
Jeff Rasley authored Sep 10, 2020

96935958
bump DSE and doc tweak · 4b1df25a
Jeff Rasley authored Sep 10, 2020

4b1df25a
only add 1bit adam reqs if mpi is installed, update cond build for cpu-adam (#400) · 240ea97b
Jeff Rasley authored Sep 10, 2020

240ea97b
update docker image and bump DSE · b29229bf
Jeff Rasley authored Sep 10, 2020

b29229bf
Website edits (#398) · 6bb5c69f
Shaden Smith authored Sep 10, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
6bb5c69f
Update README.md · 7baf3c3a
Jeff Rasley authored Sep 10, 2020

7baf3c3a

Landing page updates (#395) · a8a8b3d2

Jeff Rasley authored Sep 10, 2020


Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

a8a8b3d2

Adding sparse attention news index item (#376) · c76769c4
Arash Ashari authored Sep 10, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
c76769c4
Minjiaz/zero offload (#382) · 59ce90d0
Minjia Zhang authored Sep 10, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
59ce90d0
Sparse attention: updating code tag in documentation (#394) · be4b94be
Arash Ashari authored Sep 10, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
be4b94be
fix for 16GB v100 nodes (#393) · b1d4bd73
Jeff Rasley authored Sep 10, 2020

b1d4bd73

ZeRO tutorials (#384) · 2dea61f2

Olatunji Ruwase authored Sep 10, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

2dea61f2

Add openmpi to dockerfile · c0d5424f
Jeff Rasley authored Sep 10, 2020

c0d5424f

Fix datatype issue with sparse attention softmax (#363) · dca0b784

Jeff Rasley authored Sep 10, 2020



Fixes a dataype issue with softmax where the number of blocks being sent to the Triton kernel source was a torch.Tensor but should have been a python integer. On some environments (e.g., conda) this resulted in triton not knowing how to serialize the input (and crashing in our tests). Once switching to the correct datatype that triton expects this seems to have solved the issue.
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

dca0b784

Update documentation for 1-bit Adam (#388) · 093f09ff
Ammar Ahmad Awan authored Sep 10, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
093f09ff
Pipeline parallel training engine. (#392) · 65c2f974
Shaden Smith authored Sep 09, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
65c2f974

ZeRO-Offload release (#391) · 41db1c2f

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

09 Sep, 2020 1 commit
- Update test triggers to exclude docs · 79093d74
  Jeff Rasley authored Sep 09, 2020
  
  79093d74