Commits · 14198f2001023917854522fcbb95d9108d02e693 · OpenDAS / TransformerEngine

03 Feb, 2023 2 commits

Increase number of FP8 tensors per GEMM (#22) · 14198f20

vasunvidia authored Feb 03, 2023



* Increase number of FP8 tensors per GEMM
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Enable FP8 output tensor for fp8_gemm
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* [BERT FP8] Initial TE review comments
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Temporary fix for cuda graph non convergence
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Address review comments-2
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Review comments-3
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Cleanup
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Change for New API
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Remove unnecessary clone for D_scale, D_amax
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Avoid Roll for AMAX history size = 1
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Update onnx_te_gemm API
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Fix Lint errors
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

---------
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

14198f20

remove d2d copies (#64) · 58f19082

Kirthi Shankar Sivamani authored Feb 02, 2023



* remove d2d copies
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* cleanup
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

58f19082

31 Jan, 2023 1 commit
- Address steady memory increase and bloated checkpoints (#63) · c126396b
  Kirthi Shankar Sivamani authored Jan 31, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  c126396b
27 Jan, 2023 2 commits
- Fix the integer overflow in fused softmax (#60) · f63b27e8
  Przemyslaw Tredak authored Jan 27, 2023
```
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
```
  f63b27e8
- Add docs for FP8 calibration (#61) · b67fe451
  Kirthi Shankar Sivamani authored Jan 27, 2023
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  b67fe451
26 Jan, 2023 1 commit

Numerics fix from #40 (#59) · 677de00d

Kirthi Shankar Sivamani authored Jan 25, 2023



Stats and dgrad bug fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

677de00d

25 Jan, 2023 1 commit

Bug fixes from #40 (#58) · f985db38

Kirthi Shankar Sivamani authored Jan 25, 2023



* Fix conflict from #40
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Bug fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

f985db38

24 Jan, 2023 2 commits

Schetlur/fp8 calibration (#40) · 7fc079a4

schetlur-nv authored Jan 24, 2023



* Initial commit for fp8 calibration.
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Fixes to make unit tests pass
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

* Added test and finished implementation
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Cleaning up handling of save_for_backward in Linear
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Removing commented lines
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Minor fix to mnist test.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Moving stats computation to the forward pass instead of pre_forward, and extending to all other layers
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Pylint cleanup.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Fixing unit test failures.
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Misc changes
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>

* Fixing bad indentation from master merge and moving some code into the needs_stats conditional
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>
Signed-off-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>
Signed-off-by: Sharan Chetlur <schetlur@nvidia.com>
Signed-off-by: schetlur-nv <116769508+schetlur-nv@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <schetlur@dlcluster.nvidia.com>

7fc079a4

Add margin for LayerNorm kernel SM usage (#57) · 275902fd

Kirthi Shankar Sivamani authored Jan 24, 2023



* Add margin for LayerNorm kernel SM usage
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* revert stylistic changes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

275902fd

20 Jan, 2023 1 commit

Remove intermediate dispatch functions (#56) · 7f270330

Kirthi Shankar Sivamani authored Jan 19, 2023


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

7f270330

19 Jan, 2023 1 commit

Fix NVTX name for LN backward (#55) · f22929cc

Kirthi Shankar Sivamani authored Jan 19, 2023


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

f22929cc

18 Jan, 2023 1 commit

Add ONNX export support for TE modules (#41) · 6c9ce179

asfiyab-nvidia authored Jan 18, 2023



* Add ONNX export support for TE modules (#1)

* Add TorchScript Operators
* Add symbolic methods to ONNX exporter
* Add tests for the ONNX export
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fixes for pylint tests
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix pylint warning in softmax.py
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* move FP8 ORT lib inside tests/
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* enable cross attention tests
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* refactor code by @nzmora
* Increase layernorm FP16 threshold
* Normalize onnx file names: _ separates configs; - separates words in a single config
* Add get_attn_mask_str and fix mask string
* Add missing ONNX files
* Moved generated ONNX files to tests/gen_onnx_models/
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix merge conflict changes
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix Q/DQ scale input
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* enable FP16 config when bias is disabled
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix pylint check errors
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* updates
1. remove List import for pylint failure
2. address comments: remove state tensors from GPU
3. address comments: Update reverse_map_dtype function and add to namespace
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* minor fix: coding guidelines
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* changes:
1. skip FP8 tests on  non-hopper devices
2. minor fix for C++ lint check
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* fix onnxruntime version
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* minor fix: add space between code and comment
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* changes
1. update copyrights
2. update path to ORT .so
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>

* Apply suggestions from code review
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: asfiyab-nvidia <117682710+asfiyab-nvidia@users.noreply.github.com>
Signed-off-by: Asfiya Baig <asfiyab@nvidia.com>
Signed-off-by: asfiyab-nvidia <117682710+asfiyab-nvidia@users.noreply.github.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

6c9ce179

17 Jan, 2023 2 commits

Change version to 0.6.0dev · e2ad34e9
Przemek Tredak authored Jan 17, 2023
```
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
```
e2ad34e9

Move calculation of scale inverse to framework (#51) · 02a3582c

Kirthi Shankar Sivamani authored Jan 17, 2023



* Move scale inverse calculation to framework
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* cleanup
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix RMSNorm
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix tests
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Fix gated kernel/geglu
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

02a3582c

13 Jan, 2023 1 commit

Indexing fix for bug in virtual interleaved pipelining configs (#52) · 40467fc2

Kirthi Shankar Sivamani authored Jan 13, 2023



Virtual interleaved pipelining fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

40467fc2

12 Jan, 2023 1 commit

Add NVTX to TE modules (#50) · aadd3e7c

Przemyslaw Tredak authored Jan 12, 2023



* Add NVTX to TE modules
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix pylint
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix NVTX in _prepare_backward
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Add NVTX to C API
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix cpplint and link nvToolsExt
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Add NVTX to GeGlu
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

aadd3e7c

10 Jan, 2023 1 commit

Add GeGLU and the corresponding gradient kernels (#47) · eed1fa26

zlsh80826 authored Jan 11, 2023



* Add GeGLU and DGeGLU
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Add DGeGLUCT
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Update copyright year
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Refine shape check
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Code refine
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>

eed1fa26

09 Jan, 2023 1 commit

Add RMSNorm (#45) · 37cc3625

zlsh80826 authored Jan 10, 2023



* Add rmsnorm kernels
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Add rmsnorm cpp unit test
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Apply new Tensor struct
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Move scale/scale_inv/amax into the TE Tensor struct
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Add document
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Separate rmsnorm kernels from the layernorm
Signed-off-by: Reese Wang <rewang@nvidia.com>

* fix indent
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Update rmsnorm test cases
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Update copyright year
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Fix the support matrix on the document
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Move register macro out of utils.cuh
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>

37cc3625

05 Jan, 2023 1 commit

Enforce boolean attention mask type (#49) · 0c9c0ba1

Kirthi Shankar Sivamani authored Jan 04, 2023



* Enforce boolean attention mask type
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix tests
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

0c9c0ba1

04 Jan, 2023 1 commit

Docs: remove build warnings and add FP8 caching note (#44) · d6ff6f4d

Kirthi Shankar Sivamani authored Jan 04, 2023



* docs: remove build warnings and add FP8 caching note
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* add comment about amax history
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

d6ff6f4d

03 Jan, 2023 2 commits

Update copyright year (#48) · 64a8dc90

Przemyslaw Tredak authored Jan 03, 2023


Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

64a8dc90

Reduce unit tests time (#46) · 63822008

zlsh80826 authored Jan 04, 2023



* Use -O2 for the test_operator
Signed-off-by: Reese Wang <rewang@nvidia.com>

* Increase test parallelism
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>

63822008

17 Dec, 2022 1 commit

Fix in MHA cross attention path (#43) · c49f90d3

Kirthi Shankar Sivamani authored Dec 16, 2022



fix unfused qkv param Xattn path
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

c49f90d3

15 Dec, 2022 3 commits
- Change version to 0.5.0dev · 003f0549
  Przemek Tredak authored Dec 15, 2022
  
  003f0549
- Fix LayerNorm API param names (#42) · 0e9b2771
  Kirthi Shankar Sivamani authored Dec 15, 2022
```
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
```
  0e9b2771
- Add new users to TE CI · e456110b
  Przemek Tredak authored Dec 15, 2022
  
  e456110b
08 Dec, 2022 1 commit

Move the amax/scale/scale_inv into the TE Tensor struct. (#33) · a5ba71f3

Przemyslaw Tredak authored Dec 08, 2022



* Move the amax/scale/scale_inv into the TE Tensor struct.
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Handle multi_cast_transpose
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Changed softmax to new Tensor
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* First pass at the cpp tests
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Round of fixes
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix multi_cast_transpose
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Fix cast_to_fp8
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptrendx@gmail.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptrendx@gmail.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

a5ba71f3

07 Dec, 2022 1 commit

Ensure contiguous inputs (#38) · 509bf877

Kirthi Shankar Sivamani authored Dec 06, 2022



ensure contiguous inputs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

509bf877

06 Dec, 2022 1 commit

Softmax docstrings and type fixes (#37) · 89f94ba2

Kirthi Shankar Sivamani authored Dec 05, 2022



* Softmax docs and type fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* lint whitespace
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* change API, better naming, const fixes
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

89f94ba2

02 Dec, 2022 1 commit

Link performance optimization tutorial to docs (#36) · 0291a608

Przemyslaw Tredak authored Dec 02, 2022


Signed-off-by: Przemyslaw Tredak <ptredak@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptredak@nvidia.com>

0291a608

01 Dec, 2022 3 commits

cleanup pylintrc (#35) · 84d3d12a

Kirthi Shankar Sivamani authored Dec 01, 2022


Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

84d3d12a

Framework agnostic softmax kernels (#30) · f674d49e

Kirthi Shankar Sivamani authored Dec 01, 2022



* Make fused softmax kernels PyTorch independent
Co-authored-by: Sean Lee <selee@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Address review comments
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* move get_batch_per_block to python
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix license in softmax.h
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Co-authored-by: Sean Lee <selee@nvidia.com>

f674d49e

Add pylint to Lint action (#19) · b2743878

Przemyslaw Tredak authored Nov 30, 2022



* Add pylint to Lint action
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Test Ubuntu 20.04
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Pylint inside the container
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>

* Update transformer_engine/pytorch/distributed.py
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptrendx@gmail.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Przemyslaw Tredak <ptrendx@gmail.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

b2743878

30 Nov, 2022 1 commit

Fix illegal memory access in general layer norm backward kernel (#34) · 693d6541

Tim Moon authored Nov 30, 2022



Fix illegal memory access in layernorm backward kernel
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com>

693d6541

28 Nov, 2022 1 commit

Multi-tensor cast-transpose (#18) · 2a3916b4

Tim Moon authored Nov 28, 2022



* Add kernel for multi-tensor cast-transpose
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix incorrect test function in multi-tensor cast-transpose unit test
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Remove std::vector from multi-tensor cast-transpose function signature

Makes sure the main header is C-compatible.
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Przemyslaw Tredak <ptredak@nvidia.com>

2a3916b4

23 Nov, 2022 1 commit

Full activation recompute checkpointing bug fix (#31) · d10dfb57

Kirthi Shankar Sivamani authored Nov 23, 2022



fix checkpoint loading bug for FAR
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

d10dfb57

18 Nov, 2022 2 commits

Changed version to 0.4.0dev · 6d2294b2
Przemek Tredak authored Nov 18, 2022
```
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
```
6d2294b2

Documentation for advanced performance optimizations (#20) · 8e7f4c8c

Tim Moon authored Nov 18, 2022



* Documentation for advanced perf optimizations

Fix bug where we were doing backward passes inside fp8_autocast in example notebooks.
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Minor tweaks to advanced perf optimization docs

Review suggestions from @ptrendx
Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Rewording sequence parallelism in advanced perf optimization docs

Review suggestion from @ksivaman
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

8e7f4c8c

17 Nov, 2022 1 commit

Make amax reduction optional (#11) · 681bf9ad

Kirthi Shankar Sivamani authored Nov 16, 2022



* Make amax reduction optional
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* remove setup for global amax redux for optional case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Improve documentation
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Address documentation review
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Documentation fix
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* better FP8 checkpointing
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Making checkpointing backwards compatible
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Add deprecation warning for old checkpoint loading
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* fix checkpointing for fp8 recompute case
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* improvements to deprecation warning
Co-authored-by: Przemyslaw Tredak <ptrendx@gmail.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Co-authored-by: Przemyslaw Tredak <ptredak@nvidia.com>
Co-authored-by: Przemyslaw Tredak <ptrendx@gmail.com>

681bf9ad

16 Nov, 2022 1 commit

Fix bugs for full activation recompute in FP8 (#24) · c149c145

Kirthi Shankar Sivamani authored Nov 16, 2022



* Fix bugs for full activation recompute in FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Ensure identical numerics in recomputation for pipeline parallelism
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* expose checkpoint API and add docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* complete checkpointing docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

c149c145