- 12 Apr, 2024 2 commits
-
-
Kirthi Shankar Sivamani authored
Fix 0 grid size Signed-off-by:Kirthi Shankar Sivamani <ksivamani@nvidia.com>
-
Kirthi Shankar Sivamani authored
* FP8 cuda graphs Signed-off-by:
Vasudevan Rengasamy <vrengasamy@nvidia.com> Co-authored-by:
Charlene Yang <charleney@nvidia.com> * Fix numerics Signed-off-by:
-
- 10 Apr, 2024 1 commit
-
-
Jinze Xue authored
Signed-off-by:
Jinze Xue <jinzex@nvidia.com> Co-authored-by:
-
- 06 Apr, 2024 3 commits
-
-
Reese Wang authored
* value_and_grad requires same shape for input and gradients Signed-off-by:
Reese Wang <rewang@nvidia.com> * Use high precision layernorm Signed-off-by:
-
Sangkug Lym authored
fix the default userbuffer communicator init settings Signed-off-by:Sangkug Lym <slym@nvidia.com>
-
Jaemin Choi authored
* Enable DGRAD RS overlap Signed-off-by:
Jaemin Choi <jaeminc@nvidia.com> * fix lint; apply suggestions Signed-off-by:
-
- 04 Apr, 2024 4 commits
-
-
Sangkug Lym authored
* userbuffer fp8 reduction support for individual overlap Signed-off-by:
-
Tim Moon authored
Signed-off-by:Tim Moon <tmoon@nvidia.com>
-
Kirthi Shankar Sivamani authored
* Args can be None Signed-off-by:
-
Pavel Shamis (Pasha) authored
* Fixing potential integer overflow on sequence counter Current implementation may potential cause hangs or data corruption Signed-off-by:
Pasha (Pavel) Shamis <pasharesearch@gmail.com> * Fixing typo in comments Addressing reviewers comments Signed-off-by:
-
- 03 Apr, 2024 5 commits
-
-
Santosh Bhavani authored
* Update README.rst 1. Updated latest news with databricks blog 2. Fixed formatting issues 3. Added GTC 2024 video Signed-off-by:
Santosh Bhavani <santosh@semantic.md> * Update README.rst added back overview marker for docs generation Signed-off-by:
-
Sangkug Lym authored
* Atomic gemm for TP-AR and TP-RS overlap with P2P exchanges Signed-off-by:
-
Sangkug Lym authored
* Do not store input activations when not computing weight gradients Signed-off-by:
-
vasunvidia authored
Fix license, and sign off everything Signed-off-by:
-
Kirthi Shankar Sivamani authored
This reverts commit 965803c9.
-
- 02 Apr, 2024 1 commit
-
-
Tim Moon authored
Signed-off-by:
-
- 31 Mar, 2024 1 commit
-
-
Paweł Gadziński authored
Llama tutorial fixes - all Signed-off-by:
Pawel Gadzinski <pgadzinski@nvidia.com> Co-authored-by:
-
- 29 Mar, 2024 2 commits
-
-
Kirthi Shankar Sivamani authored
* Fix backward compatibility with checkpoint API Signed-off-by:
-
Tim Moon authored
Perform FP8 cast on gathered layernorm output in LayerNormLinear Signed-off-by:
-
- 22 Mar, 2024 2 commits
-
-
Jaemin Choi authored
* Enable TP-AG overlap with return_layernorm_output Signed-off-by:
-
Reese Wang authored
* Remove unused headers Signed-off-by:
-
- 21 Mar, 2024 2 commits
-
-
Sangkug Lym authored
* TP-RS overlap with send/recv Atomic GEMM based TP-RS overlap with send/recv Signed-off-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com>
-
Kite0011 authored
[Pytorch] Update context parallel softmax lse correction func. Signed-off-by:
kitefang <kitefang@tencent.com> Co-authored-by:
-
- 20 Mar, 2024 2 commits
-
-
Sudhakar Singh authored
* tutorial and doc fixes Signed-off-by:
Sudhakar Singh <sudhakars@nvidia.com> * remove extra code Signed-off-by:
-
Kirthi Shankar Sivamani authored
Signed-off-by:
-
- 18 Mar, 2024 1 commit
-
-
Kirthi Shankar Sivamani authored
Signed-off-by:
-
- 15 Mar, 2024 1 commit
-
-
Rachit Garg authored
* fix the perf regression because of constant property polling of the device Signed-off-by:
Rachit Garg <rachitg@nvidia.com> * Fix lint error Signed-off-by:
-
- 14 Mar, 2024 1 commit
-
-
Keshav Balasubramanian authored
* disallow sharding of layernorm learnable parameters; force duplication Signed-off-by:
Keshav <keshavb@nvidia.com> * fix tests and support tensors for gamma/beta in layernorms Signed-off-by:
-
- 13 Mar, 2024 2 commits
-
-
Santosh Bhavani authored
Update README.rst - Latest News Added an entry to Latest News section Signed-off-by: -
Rachit Garg authored
Add envvar for SM margin in GEMM Signed-off-by:
-
- 11 Mar, 2024 1 commit
-
-
Tim Moon authored
Remove deprecated cudnn_frontend::throw_if Deprecated in cudnn-frontend 1.1.0. Signed-off-by:
-
- 07 Mar, 2024 1 commit
-
-
Hongbin Liu authored
* add_dtype_for_userbuf Signed-off-by:
Hongbin Liu <hongbinl@nvidia.com> * Update transformer_engine/pytorch/module/base.py Signed-off-by:
-
- 06 Mar, 2024 3 commits
-
-
Oleg Goncharov authored
* Modified MHA and DPA logic to use causal softmax and FA for inference Signed-off-by:
Oleg Goncharov <ogoncharov@nvidia.com> * Adjusted unfused attention and softmax logic for inference Signed-off-by:
Oleg Goncharov <64355998+Oleg-Goncharov@users.noreply.github.com> * Apply suggestions from code review [sequence start-end] Co-authored-by:
Przemyslaw Tredak <ptredak@nvidia.com> Co-authored-by:
-
George Karpenkov authored
Bias and seed can both be None, type checking is failed otherwise. Signed-off-by:George Karpenkov <george@metaworld.me>
-
Chen Cui authored
* first draft of return_layernorm_output_gathered Signed-off-by:
Chen Cui <chcui@nvidia.com> * explain use case more thoroughly in docstring Signed-off-by:
Chen Cui <cxcui@alumni.cmu.edu> * Update transformer_engine/pytorch/module/layernorm_linear.py Co-authored-by:
-
- 05 Mar, 2024 2 commits
-
-
Jaemin Choi authored
Signed-off-by:
-
Zhenhuan Liu authored
Signed-off-by:Zhenhuan Liu <nkulzh16@gmail.com>
-
- 04 Mar, 2024 2 commits
-
-
Kirthi Shankar Sivamani authored
Update checkpoint API doc Signed-off-by: -
Jinze Xue authored
* Enable incremental CMake build Signed-off-by:
Jinze Xue <155670984+jinzex@users.noreply.github.com> * Update setup.py Co-authored-by:
-
- 01 Mar, 2024 1 commit
-
-
Kirthi Shankar Sivamani authored
* Avoid updating real during param cast Signed-off-by:
-
