- 30 Nov, 2023 1 commit
-
-
Ming-Xu Huang authored
Use relative idx to ScaledUpperTriangMaskedSoftmaxFwdPrimitive.abstract to support batching. Signed-off-by:Ming Huang <mingh@nvidia.com>
-
- 28 Nov, 2023 3 commits
-
-
Marks101 authored
* [PyTorch] Linear: fix computation for wgrad if sequence_parallel=True Signed-off-by:
Markus Schnoes <markus.schnoes@gmx.de> * Remove buggy gather_along_last_dim Signed-off-by:
Tim Moon <tmoon@nvidia.com> * [PyTorch] Linear: fix line length Signed-off-by:
-
Deepak Narayanan authored
Getting warnings of the following form using ToT TE: ``` /usr/local/lib/python3.10/dist-packages/transformer_engine/pytorch/attention.py:852: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() data_ptr = grad_outputs[0].storage().data_ptr() ``` Signed-off-by:Deepak Narayanan <2724038+deepakn94@users.noreply.github.com>
-
Tim Moon authored
Use unsigned char for byte dtype Signed-off-by:
-
- 23 Nov, 2023 1 commit
-
-
Tian Zheng authored
* Add async TP comm Signed-off-by:
Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Add set CUDA_DEVICE_MAX_CONNECTIONS warning for tp overlap Signed-off-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com>
-
- 21 Nov, 2023 1 commit
-
-
Shijie authored
* fix cudnn FA softmax shape Signed-off-by:
Shijie Wang <jaywan@nvidia.com> * set inplace rng_state Signed-off-by:
-
- 20 Nov, 2023 1 commit
-
-
zlsh80826 authored
* Remove assertion for NO_MASK Signed-off-by:
Reese Wang <rewang@nvidia.com> * Fix JAX distributed unit tests name Signed-off-by:
-
- 17 Nov, 2023 2 commits
-
-
cyanguwa authored
* disable FAv2.1 if causal+cross attn Signed-off-by:
Charlene Yang <8636796+cyanguwa@users.noreply.github.com> * remove comment and add warning Signed-off-by:
-
Kirthi Shankar Sivamani authored
* Delay caching of transposes Signed-off-by:
Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Review comment Signed-off-by:
-
- 16 Nov, 2023 1 commit
-
-
jon-chuang authored
add comments Signed-off-by:Jon Chuang <9093549+jon-chuang@users.noreply.github.com>
-
- 15 Nov, 2023 1 commit
-
-
cyanguwa authored
* fix condition checks related to FA head_dim Signed-off-by:
-
- 14 Nov, 2023 1 commit
-
-
Ming-Xu Huang authored
* Refactor sharding.py for the further custom_partitioning migration Signed-off-by:
Donglin Yang <dongliny@nvidia.com>
-
- 13 Nov, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
Improve PyTorch memory usage Signed-off-by: -
zlsh80826 authored
[C/JAX] Support more mask types for the arbitrary seqlen kernels and minor changes of JAX bias (#469) * Move bias to float32 Signed-off-by:
-
- 09 Nov, 2023 1 commit
-
-
Sangkug Lym authored
* Make user buffer name configurable Signed-off-by:
Sangkug Lym <slym@nvidia.com> * Apply suggestions from code review Signed-off-by:
-
- 08 Nov, 2023 2 commits
-
-
Selvaraj Anandaraj authored
* Returning an empty tensor of param dtype for wgrad Signed-off-by:
Selvaraj Anandaraj <selvaraja@computelab-frontend-4-ub22.nvidia.com> * lint Signed-off-by:
-
zlsh80826 authored
* Deprecate QKV_INTERLEAVED use in JAX Signed-off-by:
-
- 03 Nov, 2023 1 commit
-
-
Xiaowei Ren authored
fix bwd error with FA v2 Signed-off-by:
Xiaowei Ren <xren@nvidia.com> Co-authored-by:
-
- 31 Oct, 2023 1 commit
-
-
Tim Moon authored
* Experimental FP8 tensor Co-authored-by:
Sudhakar Singh <sudhakars@nvidia.com> Co-authored-by:
Przemyslaw Tredak <ptrendx@gmail.com> Signed-off-by:
-
- 24 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
* paddle documentation Signed-off-by:
-
Tim Moon authored
* Do not include logging macros in installed C headers Signed-off-by:
Jan Bielak <jbielak@nvidia.com> Signed-off-by:
-
- 23 Oct, 2023 2 commits
-
-
niboshi authored
Signed-off-by:niboshi <niboshi000@gmail.com>
-
Kirthi Shankar Sivamani authored
* initial test fix Signed-off-by:
-
- 20 Oct, 2023 4 commits
-
-
Alp Dener authored
fixed incorrect of extend_fsdp_sharding_meta() in cross_fused_attn() Signed-off-by:Alp Dener <adener@nvidia.com>
-
zlsh80826 authored
canonicalize the dtype for the better user experience Signed-off-by: -
Tim Moon authored
Signed-off-by:
-
Przemyslaw Tredak authored
* Ability to check cuDNN version from Python Signed-off-by:
Przemek Tredak <ptredak@nvidia.com> * Modify the fused attention test to not use the CUDNN_VERSION env variable which is specific to NGC containers Signed-off-by:
-
- 17 Oct, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
Improve docs Signed-off-by:
-
- 12 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
* Add class for RNG state tracker. Signed-off-by:
-
Tim Moon authored
* Debug PyTorch and Paddle tests on Ada Signed-off-by:
-
- 11 Oct, 2023 3 commits
-
-
Kirthi Shankar Sivamani authored
Signed-off-by: -
Xiaowei Ren authored
* rename set_context_parallel_running to set_context_parallel_group Signed-off-by:
-
Kirthi Shankar Sivamani authored
Inference params support Signed-off-by:
-
- 10 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
Signed-off-by: -
Kirthi Shankar Sivamani authored
Fix docs Signed-off-by:
-
- 09 Oct, 2023 1 commit
-
-
cyanguwa authored
* add support for h2d/2hd in 8.9.6 Signed-off-by:
-
- 06 Oct, 2023 1 commit
-
-
Ming-Xu Huang authored
* [JAX] Enhance Dropout in TransformerLayer. 1. Fixed missing setup of dropout RNG key in TransformerLayer and LayerNormMLP. 2. Allowing seperated dropout rate for FC1's output and other hiddens. Signed-off-by:
-
- 05 Oct, 2023 1 commit
-
-
vasunvidia authored
* Initial commit Signed-off-by:
Vasudevan Rengasamy <vrengasamy@nvidia.com> * Repro for RS output mismatch with Single GEMM + Split pipelined RS Signed-off-by:
-
- 04 Oct, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
* initial changes [wip] Signed-off-by:
-
- 03 Oct, 2023 1 commit
-
-
Frédéric Bastien authored
Signed-off-by:Frederic Bastien <fbastien@nvidia.com>
-
