- 13 Nov, 2023 3 commits
-
-
Santosh Bhavani authored
* Update README.rst - Installation section Added pip install instructions and cleaned up pre-reqs and FlashAttention-2 section Signed-off-by:
Santosh Bhavani <santosh@semantic.md> * Update README.rst Signed-off-by:
Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by:
-
Kirthi Shankar Sivamani authored
Improve PyTorch memory usage Signed-off-by: -
zlsh80826 authored
[C/JAX] Support more mask types for the arbitrary seqlen kernels and minor changes of JAX bias (#469) * Move bias to float32 Signed-off-by:
Reese Wang <rewang@nvidia.com> * Enable varlen Signed-off-by:
-
- 09 Nov, 2023 1 commit
-
-
Sangkug Lym authored
* Make user buffer name configurable Signed-off-by:
Sangkug Lym <slym@nvidia.com> * Apply suggestions from code review Signed-off-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com> * Fix duplicate argument Signed-off-by:
-
- 08 Nov, 2023 5 commits
-
-
Selvaraj Anandaraj authored
* Returning an empty tensor of param dtype for wgrad Signed-off-by:
Selvaraj Anandaraj <selvaraja@computelab-frontend-4-ub22.nvidia.com> * lint Signed-off-by:
-
zlsh80826 authored
* Deprecate QKV_INTERLEAVED use in JAX Signed-off-by:
-
Tim Moon authored
* Use FP8 tolerances in JAX FP8 tests Signed-off-by:
Tim Moon <tmoon@nvidia.com> * Programmatically compute expected floating point error Signed-off-by:
-
Tim Moon authored
Disable large test cases for PyTorch Transformer layer Signed-off-by: -
Alp Dener authored
* Fixed minor bug with DistributedConfigsHelper prematurely crashing the test for insufficient GPUs before @pytest.skip condition. Signed-off-by:
Alp Dener <adener@nvidia.com> * Update tests/jax/distributed_configs_helper.py Signed-off-by:
-
- 03 Nov, 2023 2 commits
-
-
Xiaowei Ren authored
fix bwd error with FA v2 Signed-off-by:
Xiaowei Ren <xren@nvidia.com> Co-authored-by:
-
Alp Dener authored
[JAX] Regression tests for custom ops sharding with both xmap and custom_partitioning. Coverage: - layernorm: fwd/grad, zero_centered_gamma, DP, TP_COL, DP_TP_COL - rmsnorm: fwd/grad, DP, TP_COL, DP_TP_COL - softmax: fwd/grad, SCALED, SCALED_MASKED, SCALED_UPPER_TRIANG_MASKED, DP, TP_COL, TP_ROW, DP_TP_COL, DP_TP_ROW - self_fused_attn: fwd/grad, NO_BIAS, PRE_SCALE_BIAS, POST_SCALE_BIAS, NO_MASK, CAUSAL_MASK, PADDING_MASK, DP, TP_COL, DP_TP_COL - cross_fused_attn: fwd/grad, NO_BIAS, NO_MASK, PADDING_MASK, DP, TP_COL, DP_TP_COL Signed-off-by:
-
- 31 Oct, 2023 1 commit
-
-
Tim Moon authored
* Experimental FP8 tensor Co-authored-by:
Sudhakar Singh <sudhakars@nvidia.com> Co-authored-by:
Przemyslaw Tredak <ptrendx@gmail.com> Signed-off-by:
-
- 26 Oct, 2023 1 commit
-
-
zlsh80826 authored
Inherit the existing XLA_FLAGS Signed-off-by:
-
- 24 Oct, 2023 3 commits
-
-
Frédéric Bastien authored
Signed-off-by:Frederic Bastien <fbastien@nvidia.com>
-
Kirthi Shankar Sivamani authored
* paddle documentation Signed-off-by:
-
Tim Moon authored
* Do not include logging macros in installed C headers Signed-off-by:
Jan Bielak <jbielak@nvidia.com> Signed-off-by:
-
- 23 Oct, 2023 2 commits
-
-
niboshi authored
Signed-off-by:niboshi <niboshi000@gmail.com>
-
Kirthi Shankar Sivamani authored
* initial test fix Signed-off-by:
-
- 20 Oct, 2023 4 commits
-
-
Alp Dener authored
fixed incorrect of extend_fsdp_sharding_meta() in cross_fused_attn() Signed-off-by: -
zlsh80826 authored
canonicalize the dtype for the better user experience Signed-off-by: -
Tim Moon authored
Signed-off-by:
-
Przemyslaw Tredak authored
* Ability to check cuDNN version from Python Signed-off-by:
Przemek Tredak <ptredak@nvidia.com> * Modify the fused attention test to not use the CUDNN_VERSION env variable which is specific to NGC containers Signed-off-by:
-
- 19 Oct, 2023 1 commit
-
-
Kirthi Shankar Sivamani authored
RM unused docs Signed-off-by:
-
- 17 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
Signed-off-by: -
Kirthi Shankar Sivamani authored
Improve docs Signed-off-by:
-
- 13 Oct, 2023 2 commits
-
-
Tim Moon authored
Signed-off-by: -
Tim Moon authored
Signed-off-by:
-
- 12 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
* Add class for RNG state tracker. Signed-off-by:
-
Tim Moon authored
* Debug PyTorch and Paddle tests on Ada Signed-off-by:
-
- 11 Oct, 2023 3 commits
-
-
Kirthi Shankar Sivamani authored
Signed-off-by: -
Xiaowei Ren authored
* rename set_context_parallel_running to set_context_parallel_group Signed-off-by:
-
Kirthi Shankar Sivamani authored
Inference params support Signed-off-by:
-
- 10 Oct, 2023 2 commits
-
-
Kirthi Shankar Sivamani authored
Signed-off-by: -
Kirthi Shankar Sivamani authored
Fix docs Signed-off-by:
-
- 09 Oct, 2023 3 commits
-
-
cyanguwa authored
* add support for h2d/2hd in 8.9.6 Signed-off-by:
Charlene Yang <8636796+cyanguwa@users.noreply.github.com> * cull unit tests in fused_attn.py and add skipif for layout tests Signed-off-by:
-
Tim Moon authored
Signed-off-by: -
Tim Moon authored
Signed-off-by:
-
- 06 Oct, 2023 2 commits
-
-
Przemyslaw Tredak authored
Signed-off-by: -
Ming-Xu Huang authored
* [JAX] Enhance Dropout in TransformerLayer. 1. Fixed missing setup of dropout RNG key in TransformerLayer and LayerNormMLP. 2. Allowing seperated dropout rate for FC1's output and other hiddens. Signed-off-by:
Ming Huang <mingh@nvidia.com> * Fix wrong fp8 scale in _update_fp8_metas_impl Signed-off-by:
-
- 05 Oct, 2023 1 commit
-
-
vasunvidia authored
* Initial commit Signed-off-by:
Vasudevan Rengasamy <vrengasamy@nvidia.com> * Repro for RS output mismatch with Single GEMM + Split pipelined RS Signed-off-by:
-
