Commits · e2dfcadf3b30437a6232f6c689e695aa44931a4c · OpenDAS / deepspeed

24 Feb, 2021 1 commit

Fix the bias-add and add the layer-norm-eps parameter (#791) · e2dfcadf

Reza Yazdani authored Feb 24, 2021

* fix the bias-add precision and indexing and also adding the layer-norm-eps as a configurable parameter for transformer

* add ACC_HALF config

* use defined to check if ACC_Half is defined

e2dfcadf

18 Feb, 2021 1 commit
- Fix transformer kernel CUDA illegal memory access error (#765) · 1fcc5f7a
  Conglong Li authored Feb 17, 2021
  
  1fcc5f7a
13 Jan, 2021 1 commit

Move workspace memory-allocation to PyTorch (#661) · 981bc7d4

Reza Yazdani authored Jan 12, 2021

* move workspace memory-allocation to PyTorch

* refine the code based on the comments

* remove unnecessary options

* remove bsz from set_seq_len function

981bc7d4

17 Dec, 2020 1 commit
- Transformer-kernel - supporting any arbitrary sequence-length (#587) · fd2f970b
  Reza Yazdani authored Dec 17, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  fd2f970b
01 Oct, 2020 1 commit

Use parentesis around min and max to enable Windows build (#449) · 95575579

Bruno authored Oct 01, 2020



* Towards Windows build

* formatting
Co-authored-by: Bruno Cabral <bruno@potelo.com.br>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>

95575579

22 Sep, 2020 1 commit
- support dynamic sequence length in transformer kernels (#424) · f0f2a702
  RezaYazdaniAminabadi authored Sep 21, 2020
```
Co-authored-by: Conglong Li <conglong.li@gmail.com>
```
  f0f2a702
21 Sep, 2020 1 commit
- Add configurable intermediate size to transformer kernels (#423) · a148bd33
  RezaYazdaniAminabadi authored Sep 21, 2020
  
  a148bd33
11 Sep, 2020 2 commits
- Revert "supporting different intermediate sizes other than 4 * hidden_dim (#389)" (#404) · 4ac9bf60
  Jeff Rasley authored Sep 11, 2020
```
This reverts commit e549be60.
```
  4ac9bf60
- supporting different intermediate sizes other than 4 * hidden_dim (#389) · e549be60
  RezaYazdaniAminabadi authored Sep 11, 2020
```
* supporting different intermediate sizes other than 4*hidden_dim

* run precommit

* uncommnet the unit tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  e549be60
29 May, 2020 1 commit

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991