Commits · 06486a006c45cca3571151d98377a0899ddf38e8 · OpenDAS / TransformerEngine

16 Mar, 2023 1 commit
- Relax dimension checks for fp8 exec (#106) · d1d00b3e
  Kirthi Shankar Sivamani authored Mar 16, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  d1d00b3e
14 Mar, 2023 1 commit

Catch FP8 modulo16 error before cublas and fp8 kernels (#97) · d74ee5b5

Kirthi Shankar Sivamani authored Mar 13, 2023



* Catch FP8 modulo16 error before cublas and fp8 kernels
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* annotate
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

d74ee5b5

25 Feb, 2023 1 commit
- fix bug in non-FP8 nvfuser path (#81) · 67114f9e
  Kirthi Shankar Sivamani authored Feb 24, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  67114f9e
24 Feb, 2023 1 commit

Remove redundant AR for SP case (#79) · d8a2f352

Kirthi Shankar Sivamani authored Feb 23, 2023



* Remove redundant amax AR for SP case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* update advanced docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

d8a2f352

23 Feb, 2023 1 commit

Support arbitrary output dtypes in PyT GEMM functions (#75) · 5898702e

Tim Moon authored Feb 23, 2023



* Deprecate fp32_output option for PyT linear layers

Automatically detect dtype for user-provided output tensors.
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Remove deprecated options
Signed-off-by: Tim Moon <tmoon@nvidia.com>

---------
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

5898702e

16 Feb, 2023 1 commit

Sequence-parallel amax reduction fix (#74) · f06e2d85

Kirthi Shankar Sivamani authored Feb 16, 2023



* Fix no reduce_amax option for SP case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* add warning about overriding reduce_amax
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

f06e2d85

15 Feb, 2023 1 commit

Zero-centered gamma support in LayerNorm (LayerNorm1p) (#67) · 7324fe2b

Przemyslaw Tredak authored Feb 14, 2023



* C++ implementation of LayerNorm1P
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Expose zero centered gamma to pyTorch
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix ONNX export
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix ONNX export and tests
Signed-off-by: Przemyslaw Tredak <ptredak@nvidia.com>

* Fix lint
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix backward handling - C++ part
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix for backward - Python side
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix FP8 path
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Reenable the pylint check
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix the NVTX marker
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Change in the bwd kernel
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

---------
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptredak@nvidia.com>

7324fe2b

13 Feb, 2023 1 commit

Gradient enablement bug fix (#72) · 2f643ada

Kirthi Shankar Sivamani authored Feb 13, 2023



fix use of PyTorch training flag
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

2f643ada

10 Feb, 2023 1 commit

QKV parameters unfused path fixes and optimization (#66) · 04490337

Kirthi Shankar Sivamani authored Feb 10, 2023



* Bug fixes from PR 22
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Add FP8 tests to ci
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Better QKV parameter fusion
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* small fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* keep original param for unfused case to retain externally set attrs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* lint fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix ONNX exports
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* improve arg naming
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* No need to set data pointers
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* lint
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Assert memory loc in NoopCat
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Handle case of different memory in param and buffer
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix assert always true
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Reassign params memory to avoid more concats
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

04490337

07 Feb, 2023 1 commit

Bug fixes from PR 22 (#65) · 78b4e933

Kirthi Shankar Sivamani authored Feb 06, 2023



* Bug fixes from PR 22
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Add FP8 tests to ci
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* bundle unittests for ci
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

78b4e933

03 Feb, 2023 1 commit

Increase number of FP8 tensors per GEMM (#22) · 14198f20

vasunvidia authored Feb 03, 2023



* Increase number of FP8 tensors per GEMM
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Enable FP8 output tensor for fp8_gemm
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* [BERT FP8] Initial TE review comments
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Temporary fix for cuda graph non convergence
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Address review comments-2
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Review comments-3
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Cleanup
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Change for New API
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Remove unnecessary clone for D_scale, D_amax
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Avoid Roll for AMAX history size = 1
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Update onnx_te_gemm API
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Fix Lint errors
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

---------
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

14198f20

31 Jan, 2023 1 commit
- Address steady memory increase and bloated checkpoints (#63) · c126396b
  Kirthi Shankar Sivamani authored Jan 31, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  c126396b
26 Jan, 2023 1 commit

Numerics fix from #40 (#59) · 677de00d

Kirthi Shankar Sivamani authored Jan 25, 2023



Stats and dgrad bug fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

677de00d

25 Jan, 2023 1 commit

Bug fixes from #40 (#58) · f985db38

Kirthi Shankar Sivamani authored Jan 25, 2023



* Fix conflict from #40
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

f985db38

24 Jan, 2023 2 commits

Schetlur/fp8 calibration (#40) · 7fc079a4

schetlur-nv authored Jan 24, 2023



* Initial commit for fp8 calibration.
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Fixes to make unit tests pass
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Added test and finished implementation
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Cleaning up handling of save_for_backward in Linear
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Removing commented lines
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Minor fix to mnist test.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Moving stats computation to the forward pass instead of pre_forward, and extending to all other layers
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Fixing unit test failures.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Misc changes
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Fixing bad indentation from master merge and moving some code into the needs_stats conditional
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>
Signed-off-by: schetlur-nv <116769508+schetlur-nv@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

7fc079a4

Add margin for LayerNorm kernel SM usage (#57) · 275902fd

Kirthi Shankar Sivamani authored Jan 24, 2023



* Add margin for LayerNorm kernel SM usage
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* revert stylistic changes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

275902fd

18 Jan, 2023 1 commit

Add ONNX export support for TE modules (#41) · 6c9ce179

asfiyab-nvidia authored Jan 18, 2023



* Add ONNX export support for TE modules (#1)

* Add TorchScript Operators
* Add symbolic methods to ONNX exporter
* Add tests for the ONNX export
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fixes for pylint tests
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix pylint warning in softmax.py
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* move FP8 ORT lib inside tests/
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* enable cross attention tests
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* refactor code by @nzmora
* Increase layernorm FP16 threshold
* Normalize onnx file names: _ separates configs; - separates words in a single config
* Add get_attn_mask_str and fix mask string
* Add missing ONNX files
* Moved generated ONNX files to tests/gen_onnx_models/
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix merge conflict changes
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix Q/DQ scale input
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* enable FP16 config when bias is disabled
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix pylint check errors
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* updates
1. remove List import for pylint failure
2. address comments: remove state tensors from GPU
3. address comments: Update reverse_map_dtype function and add to namespace
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* minor fix: coding guidelines
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* changes:
1. skip FP8 tests on  non-hopper devices
2. minor fix for C++ lint check
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix onnxruntime version
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* minor fix: add space between code and comment
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* changes
1. update copyrights
2. update path to ORT .so
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* Apply suggestions from code review
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: asfiyab-nvidia <117682710+asfiyab-nvidia@users.noreply.github.com>
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>
Signed-off-by: asfiyab-nvidia <117682710+asfiyab-nvidia@users.noreply.github.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

6c9ce179

17 Jan, 2023 1 commit

Move calculation of scale inverse to framework (#51) · 02a3582c

Kirthi Shankar Sivamani authored Jan 17, 2023



* Move scale inverse calculation to framework
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* cleanup
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix RMSNorm
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix tests
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix gated kernel/geglu
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

02a3582c

13 Jan, 2023 1 commit

Indexing fix for bug in virtual interleaved pipelining configs (#52) · 40467fc2

Kirthi Shankar Sivamani authored Jan 13, 2023



Virtual interleaved pipelining fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

40467fc2

12 Jan, 2023 1 commit

Add NVTX to TE modules (#50) · aadd3e7c

Przemyslaw Tredak authored Jan 12, 2023



* Add NVTX to TE modules
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix pylint
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix NVTX in _prepare_backward
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Add NVTX to C API
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix cpplint and link nvToolsExt
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Add NVTX to GeGlu
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

aadd3e7c

03 Jan, 2023 1 commit

Update copyright year (#48) · 64a8dc90

Przemyslaw Tredak authored Jan 03, 2023


Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

64a8dc90

15 Dec, 2022 1 commit

Fix LayerNorm API param names (#42) · 0e9b2771

Kirthi Shankar Sivamani authored Dec 15, 2022


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

0e9b2771

07 Dec, 2022 1 commit

Ensure contiguous inputs (#38) · 509bf877

Kirthi Shankar Sivamani authored Dec 06, 2022



ensure contiguous inputs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

509bf877

17 Nov, 2022 1 commit

Make amax reduction optional (#11) · 681bf9ad

Kirthi Shankar Sivamani authored Nov 16, 2022



* Make amax reduction optional
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* remove setup for global amax redux for optional case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Improve documentation
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Address documentation review
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Documentation fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* better FP8 checkpointing
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Making checkpointing backwards compatible
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Add deprecation warning for old checkpoint loading
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix checkpointing for fp8 recompute case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* improvements to deprecation warning
Co-authored-by: Przemyslaw Tredak <ptrendx@gmail.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Co-authored-by: Przemyslaw Tredak <ptredak@nvidia.com>
Co-authored-by: Przemyslaw Tredak <ptrendx@gmail.com>

681bf9ad

16 Nov, 2022 1 commit

Fix bugs for full activation recompute in FP8 (#24) · c149c145

Kirthi Shankar Sivamani authored Nov 16, 2022



* Fix bugs for full activation recompute in FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Ensure identical numerics in recomputation for pipeline parallelism
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* expose checkpoint API and add docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* complete checkpointing docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

c149c145

15 Nov, 2022 1 commit

Fixes #26 (#29) · 9f7b0255

Kirthi Shankar Sivamani authored Nov 15, 2022



addressed LayerNormMLP bias issue #26
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

9f7b0255

03 Nov, 2022 1 commit

Conditional wgrad support (#21) · 78210127

schetlur-nv authored Nov 03, 2022



* Conditional dgrad/wgrad support
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Fixing the change to depend only on requires_grad. Also updating LayerNorm MLP
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Minor fixes.
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Adding conditional wgrad for LayerNormLinear
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* bug fix and remove conditional dgrad

Co-authored-by: schetlur-nv schetlur@nvidia.com
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Adding unit test for wgrad disabled path
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Adding more unit tests for wgrad disabled path
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Adding unit tests for fp8 wgrad disabling, and cleaning up the code.
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* fix lint errors
Co-Authored-By: Sharan Chetlur <schetlur@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Co-authored-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

78210127

04 Oct, 2022 2 commits

fix TP case to rm redundant AR (#3) · 5612ba78

Kirthi Shankar Sivamani authored Oct 04, 2022


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

5612ba78

fp8_autocast bug fix when switching from non-fp8 execution (#2) · a2caec1e

Kirthi Shankar Sivamani authored Oct 04, 2022


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

a2caec1e

28 Sep, 2022 1 commit

Inital code drop · 996ea169

Przemek Tredak authored Sep 27, 2022


Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

996ea169