Commits · 54e46e21a989ac2808e29ce3c20275738d506afe · OpenDAS / TransformerEngine

13 Nov, 2023 1 commit
- [PyTorch] Improve memory usage in backward of LayerNormLinear and LayerNormMLP (#509) · a9cfbfd3
  Kirthi Shankar Sivamani authored Nov 13, 2023
```
Improve PyTorch memory usage
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  a9cfbfd3
12 Oct, 2023 1 commit

Tim Moon authored Oct 12, 2023



* Debug PyTorch and Paddle tests on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Only run Paddle layer tests with cuDNN fMHA on supported archs
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug PyTorch fMHA tests
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Reduce JAX FP8 GEMM sizes

Avoid split-k kernels on Ada.
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Disable JAX fused self-attention test on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Run supported fused attention tests on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Run supported fused attention JAX tests on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Enable Paddle fused attention on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update reference scale calculation in TensorFlow test
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Restore backend support to reference FP8 attention impl in PyT test

Review suggestion from @cyanguwa
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix merge conflicts
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug Paddle tests on Ada
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Loosen tolerances for Paddle attention tests
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Assume causal mask implies equal seqlens in Paddle attention tests
Signed-off-by: Tim Moon <tmoon@nvidia.com>

---------
Signed-off-by: Tim Moon <tmoon@nvidia.com>

4ae34765

13 Jul, 2023 1 commit

Catch cublas FP8 errors (#317) · 8c3110d1

Kirthi Shankar Sivamani authored Jul 12, 2023



* Better dimension assert for FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* line
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

8c3110d1

25 May, 2023 1 commit

Clearer error messages for dtype and shape assertions (#245) · 871fdf51

Carlos Mocholí authored May 24, 2023



* Clearer error messages for dtype and shape assertions
Signed-off-by: Carlos Mocholí <carlossmocholi@gmail.com>

* Update transformer_engine/pytorch/utils.py
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Signed-off-by: Carlos Mocholí <carlossmocholi@gmail.com>

* Update transformer_engine/pytorch/utils.py
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Carlos Mocholí <carlossmocholi@gmail.com>

---------
Signed-off-by: Carlos Mocholí <carlossmocholi@gmail.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

871fdf51

16 Mar, 2023 1 commit
- Relax dimension checks for fp8 exec (#106) · d1d00b3e
  Kirthi Shankar Sivamani authored Mar 16, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  d1d00b3e
14 Mar, 2023 1 commit

Catch FP8 modulo16 error before cublas and fp8 kernels (#97) · d74ee5b5

Kirthi Shankar Sivamani authored Mar 13, 2023



* Catch FP8 modulo16 error before cublas and fp8 kernels
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* annotate
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

d74ee5b5

07 Mar, 2023 1 commit

Fix flash attention (#84) · 37a12c4e

Kirthi Shankar Sivamani authored Mar 07, 2023



* ignore self attention mask for causal type
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* further relax checks to run FA, update docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix pytorch softmax path
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* minimum ampere requirement for fa
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

37a12c4e

02 Mar, 2023 1 commit

Fix unfused QKV params case; stack vs interleave option (#83) · 7d6c1d02

Kirthi Shankar Sivamani authored Mar 02, 2023



* fix qkv weight unfused path
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix non FA non interleaved case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

7d6c1d02

10 Feb, 2023 1 commit

Fix gradients when using AMP (#70) · eda8f461

Kirthi Shankar Sivamani authored Feb 10, 2023



retain grad related attrs while casting
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

eda8f461

03 Jan, 2023 1 commit

Update copyright year (#48) · 64a8dc90

Przemyslaw Tredak authored Jan 03, 2023


Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

64a8dc90

28 Sep, 2022 1 commit

Inital code drop · 996ea169

Przemek Tredak authored Sep 27, 2022


Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

996ea169