Commits · c78c29f9384ec8a3addb049504a4167508fe1e4d · OpenDAS / deepspeed

01 Dec, 2020 1 commit

supporting different hidden dimensions (#559) · c78c29f9

Reza Yazdani authored Dec 01, 2020



* supporting different hidden dimensions

* add support for larger hidden dimensions (greater than 8K)

* remove empty line

* add loop unrolling factor for dropout kernels

* update different kernels based on the reviews
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

c78c29f9

12 Nov, 2020 1 commit

DeepSpeed JIT op + PyPI support (#496) · 31f46fee

Jeff Rasley authored Nov 12, 2020


Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>

31f46fee

05 Nov, 2020 1 commit

Fixing CPU-Adam convergence issue (#503) · 7d4d742b

Reza Yazdani authored Nov 05, 2020

* fixing cpu-adam

* fixing copy with optimizer for data and model parallelism

* fixing cpu-adam

* fix cpu-adam

* fix cpu-adam

7d4d742b

30 Oct, 2020 2 commits
- fixing the AVX_256 compatibility (#497) · 4c37d705
  Reza Yazdani authored Oct 30, 2020
  
  4c37d705
- Add CPUAdam optimizer for zero-offload in deepspeed engine (#484) · f5aa2547
  Reza Yazdani authored Oct 30, 2020
```
* add adamW to CPU-ADAM implementation

* supporting cpu-adam optimizer for zero-offload on deepspeed side

* bump DSE to match cpu-adam updates
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  f5aa2547
01 Oct, 2020 1 commit

Use parentesis around min and max to enable Windows build (#449) · 95575579

Bruno authored Oct 01, 2020



* Towards Windows build

* formatting
Co-authored-by: Bruno Cabral <bruno@potelo.com.br>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>

95575579

22 Sep, 2020 1 commit
- support dynamic sequence length in transformer kernels (#424) · f0f2a702
  RezaYazdaniAminabadi authored Sep 21, 2020
```
Co-authored-by: Conglong Li <conglong.li@gmail.com>
```
  f0f2a702
21 Sep, 2020 1 commit
- Add configurable intermediate size to transformer kernels (#423) · a148bd33
  RezaYazdaniAminabadi authored Sep 21, 2020
  
  a148bd33
11 Sep, 2020 2 commits
- Revert "supporting different intermediate sizes other than 4 * hidden_dim (#389)" (#404) · 4ac9bf60
  Jeff Rasley authored Sep 11, 2020
```
This reverts commit e549be60.
```
  4ac9bf60
- supporting different intermediate sizes other than 4 * hidden_dim (#389) · e549be60
  RezaYazdaniAminabadi authored Sep 11, 2020
```
* supporting different intermediate sizes other than 4*hidden_dim

* run precommit

* uncommnet the unit tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  e549be60
10 Sep, 2020 1 commit

ZeRO-Offload release (#391) · 41db1c2f

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

02 Sep, 2020 1 commit

Sparse attn + ops/runtime refactor + v0.3.0 (#343) · e5bbc2e5

Jeff Rasley authored Sep 01, 2020



* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

e5bbc2e5

29 May, 2020 1 commit

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991

16 Apr, 2020 1 commit
- Delete tmp · 675d73e0
  Jeff Rasley authored Apr 15, 2020
  
  675d73e0
03 Feb, 2020 2 commits
- Add files via upload · 7e7b0a8d
  Samyam Rajbhandari authored Feb 03, 2020
```
Lamb CUDA Kernels
```
  7e7b0a8d
- add tmp file · c04ae78a
  Jeff Rasley authored Feb 03, 2020
  
  c04ae78a