- 07 Sep, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
* Initial setup Signed-off-by:
Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix testfile Signed-off-by:
-
- 06 Sep, 2023 1 commit
-
-
Tian Zheng authored
* Add recompute Signed-off-by:
Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Support recompute core attention Signed-off-by:
-
- 01 Sep, 2023 1 commit
-
-
Tian Zheng authored
* Add control of attention dropout and hidden dropout RNG state Signed-off-by:
-
- 26 Aug, 2023 2 commits
-
-
Tian Zheng authored
* [Paddle] Add TP, DP, PP, FSDP Signed-off-by:
-
Kirthi Shankar Sivamani authored
* API change and some test fixes Signed-off-by:
-
- 25 Aug, 2023 2 commits
-
-
zlsh80826 authored
* Fused attention kernel only supports sm80 and sm90 Signed-off-by:
Reese Wang <rewang@nvidia.com> * Update transformer_engine/jax/csrc/modules.cpp Co-authored-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com> Signed-off-by:
-
cyanguwa authored
fix rng_state issue and minor compiler warning Signed-off-by:Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
-
- 19 Aug, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
* PyTorch MultiheadAttention API Signed-off-by:
-
- 17 Aug, 2023 1 commit
-
-
Shijie authored
* Add nn.layer: softmax, attention, transformer Signed-off-by:
Shijie Wang <jaywan@nvidia.com> * code refactor Signed-off-by:
-
- 16 Aug, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
* Initial refactor Signed-off-by:
-
- 11 Aug, 2023 2 commits
-
-
cyanguwa authored
* miscellenous fixes Signed-off-by:
-
Neta Zmora authored
* Add dynamically shaped input mask in test_export_softmax * Fix test_softmax_mask_fn - use env. var `NVTE_ONNX_KVCACHE_MAX_SEQ_LEN` to control whether the test uses the default mask generation function or dynamic TRILU mask slicing. * Change core_attention ONNX export test: use "no_mask" as attn mask type when testing `te.attention.DotProductAttention` w/o masking. * Use ORT CUDA backend by default. Signed-off-by:Neta Zmora <nzmora@nvidia.com>
-
- 10 Aug, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
Signed-off-by:
-
- 08 Aug, 2023 1 commit
-
-
Przemyslaw Tredak authored
Fix for the RMSNorm tests/doc/ONNX export to match the actual implementation Signed-off-by:Przemek Tredak <ptredak@nvidia.com>
-
- 07 Aug, 2023 1 commit
-
-
zlsh80826 authored
* Fix flash attention dropout probability with inference Signed-off-by:
-
- 03 Aug, 2023 1 commit
-
-
Ming-Xu Huang authored
* Cast Flax collections to FrozenDict as WAR to adapt Flax 0.7.1. Signed-off-by:
Ming Huang <mingh@nvidia.com> * Adding min version of flax to requirements.txt in examples Signed-off-by:
Tim Moon <tmoon@nvidia.com> --------- Signed-off-by:
-
- 01 Aug, 2023 1 commit
-
-
Tian Zheng authored
* Add FP8 support - Add FP8 recipe - Add FP8 path for nn layers - Add MNIST FP8 example Signed-off-by:
-
- 29 Jul, 2023 1 commit
-
-
cyanguwa authored
* add support for multi-query/grouped-query attention Signed-off-by:
-
- 27 Jul, 2023 1 commit
-
-
Przemyslaw Tredak authored
* Exposing RMSNorm in pyTorch extensions Signed-off-by:
Przemyslaw Tredak <ptrendx@gmail.com> Signed-off-by:
-
- 20 Jul, 2023 2 commits
-
-
Shijie authored
* add flash attn tests Signed-off-by:
-
Tim Moon authored
Replace deprecated sharding API Signed-off-by:
-
- 19 Jul, 2023 1 commit
-
-
Tian Zheng authored
* Add Linear layer (FP16) Signed-off-by:
-
- 18 Jul, 2023 1 commit
-
-
zlsh80826 authored
* Fully remove attn_type and set self_attn_mask_type default to 'causal' Signed-off-by:
-
- 15 Jul, 2023 1 commit
-
-
Tim Moon authored
* Disable TorchDynamo optimizations in PyTorch modules Signed-off-by:
-
- 14 Jul, 2023 1 commit
-
-
cyanguwa authored
* Fix bprop for cuDNN 8.9.3 Signed-off-by:
-
- 13 Jul, 2023 1 commit
-
-
Neta Zmora authored
* Fix FP32 LayerNorm ONNX export When running inference use a fwd method that is registered with torchscript. Signed-off-by:
-
- 07 Jul, 2023 1 commit
-
-
Ming-Xu Huang authored
Signed-off-by:
-
- 06 Jul, 2023 1 commit
-
-
Shijie authored
* add more ops Signed-off-by:
-
- 26 Jun, 2023 1 commit
-
-
galagam authored
Signed-off-by:
Gal Hubara Agam <ghubaraagam@nvidia.com> Co-authored-by:
-
- 22 Jun, 2023 2 commits
-
-
Tian Zheng authored
* Add cast_transpose Add gelu, gelu_fp8 Add cast_transpose_bgrad_dgelu Add layernorm_fwd and layernorm_fwd_fp8 Add layernorm_bwd Signed-off-by:
-
cyanguwa authored
* add long sequence support and unify three backends for fused attention Signed-off-by:
-
- 20 Jun, 2023 3 commits
-
-
galagam authored
Same filename is used for all activations, file is getting overridden to last activation type Signed-off-by:
-
zlsh80826 authored
* Enable fused attention dropout Signed-off-by:
-
zlsh80826 authored
* Add self_attn_mask_type and replace attn_type Signed-off-by:
-
- 16 Jun, 2023 2 commits
-
-
Neta Zmora authored
* Fix softmax ONNX export * BF16 is validated using "fake i/o": ie. instead of using BF16 as input/output, use FP32 input/output and convert to/from BF16 in the forward method. * Wrap softmax symbolic functions with conversion to/from FP32 to produce the same semantics as TE's softmax (compute is performed at FP32 precision regardless of input/output data type). Signed-off-by:
-
Kirthi Shankar Sivamani authored
* ReLU ONNX export Signed-off-by:
-
- 13 Jun, 2023 2 commits
-
-
Przemyslaw Tredak authored
* Added ReLU and GLU variants to common Signed-off-by:
-
zlsh80826 authored
Move jax.experimental.maps.Mesh to jax.sharding.Mesh Signed-off-by:
-
- 08 Jun, 2023 1 commit
-
-
Kaixi Hou authored
* Only use one gpu for tensorflow tests Signed-off-by:
kaixih <kaixih@nvidia.com> * Simplify the change Signed-off-by:
-
- 07 Jun, 2023 1 commit
-
-
Frédéric Bastien authored
* Use the same default in the function to what the class default. Signed-off-by:
Frederic Bastien <fbastien@nvidia.com> * Assert instead of silently ignoring not supported variation. Small doc correction, amax_compute_algo is partially supported. Signed-off-by:
Frédéric Bastien <frederic.bastien@gmail.com> * grammar Signed-off-by:
-
