- 25 Aug, 2025 2 commits
-
-
Phuong Nguyen authored
* added shardy warning Signed-off-by:
Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by:
-
Ace Eldeib authored
Signed-off-by:
Ace Eldeib <alexeldeib@gmail.com> Co-authored-by:
Xin Yao <xiny@nvidia.com>
-
- 23 Aug, 2025 1 commit
-
-
Tim Moon authored
* Return dummy wgrad tensors when requested by Mcore Signed-off-by:
Tim Moon <tmoon@nvidia.com> * Apply suggestions from code review Co-authored-by:
Jan Bielak <janekb04@icloud.com> Signed-off-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com> --------- Signed-off-by:
-
- 22 Aug, 2025 1 commit
-
-
Phuong Nguyen authored
update NGC version Signed-off-by:
-
- 21 Aug, 2025 1 commit
-
-
Md Fahim Faysal Khan authored
* added cp strategy arg to DPA api Signed-off-by:
Md Fahim Faysal Khan <mdfahimfaysa@nvidia.com> * converted DPA cp_strategy to string Signed-off-by:
-
- 20 Aug, 2025 6 commits
-
-
Tim Moon authored
Signed-off-by:
-
Tim Moon authored
* Fix incorrect version checks for atomic GEMM Signed-off-by:
pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
-
Tim Moon authored
Avoid garbage collection when capturing a CUDA Graph Signed-off-by: -
jberchtold-nvidia authored
[JAX] Error checking for mesh resource and update GemmPrimitive to use global_mesh_resource().fsdp_resource (#2088) * Enforce global MeshResource is set Signed-off-by:
Jeremy Berchtold <jberchtold@nvidia.com> * Use global_mesh_resource().fsdp_resource in gemm primitive Signed-off-by:
-
Paweł Gadziński authored
* code drop Signed-off-by:
Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by:
-
Jan Bielak authored
* Add rmsnorm_bwd_add Signed-off-by:
Jan Bielak <jbielak@nvidia.com> * Add BackwardAddRMSNorm fused operation Signed-off-by:
-
- 19 Aug, 2025 1 commit
-
-
Kirthi Shankar Sivamani authored
Update trigger-ci.yml Signed-off-by:Kirthi Shankar Sivamani <ksivamani@nvidia.com>
-
- 18 Aug, 2025 5 commits
-
-
Przemek Tredak authored
Signed-off-by:Przemek Tredak <ptredak@nvidia.com>
-
Phuong Nguyen authored
* fix fsdp Signed-off-by:
-
Xin Yao authored
* check if the given recipe is supported in fp8_autocast Signed-off-by:
-
Tim Moon authored
* Update list of authorized CI users Signed-off-by:
-
jberchtold-nvidia authored
Fix flax variables when creating quantizers directly from a recipe Signed-off-by:
-
- 16 Aug, 2025 1 commit
-
-
jomitchellnv authored
fix: fixes multi head attention for context parallel: rotary embedding to use padded cu_seq_lens (#2077) fix: fixes mha to use padded cu_seq_lens during cp Signed-off-by:Jonathan Mitchell <jomitchell@nvidia.com>
-
- 15 Aug, 2025 2 commits
-
-
Jan Bielak authored
* Add `nvte_cublas_gemm_scaled` Signed-off-by:
-
Kshitij Lakhani authored
* Move some dist fused attn tests to L2 1. TestReorderCausalLoadBalancing: Run two (non symmetric) BSHD/SBHD data shape combination 2. TestDistributedSelfAttn: Run only one (smaller) BSHD type data shape combination 3. TestDistributedCrossAttn: Run only one (smaller) BSHD type data shape combination 4. TestDistributedContextParallelSelfAttn: Run all cp1 combinations Signed-off-by:
Kshitij Janardan Lakhani <klakhani@nvidia.com> * Use pytest_parametrize_wrapper for splitting fused attn distributed JAX tests as L1 and L2 Signed-off-by:
Kshitij Janardan Lakhani <klakhani@login-preos01.a51.clusters.nvidia.com> * Remove test code for L2 dist residing in L2 test.sh Signed-off-by:
-
- 14 Aug, 2025 6 commits
-
-
Kirthi Shankar Sivamani authored
Add launch bounds to swizzle kernel, use empty scale inv Signed-off-by: -
Tim Moon authored
* Unfused impl for dbias-quantize Signed-off-by:
-
Tim Moon authored
Avoid registering FP8 recipe update in ops without backward pass Signed-off-by:
-
Tim Moon authored
* Register weight/bias params in linear op Signed-off-by:
-
Xin Yao authored
* reduce driver calls Signed-off-by:
-
Kshiteej K authored
* fix: update grad_output quant to avoid redundant work Signed-off-by:
kshitij12345 <kshitijkalambarkar@gmail.com> * add test Signed-off-by:
-
- 13 Aug, 2025 6 commits
-
-
Phuong Nguyen authored
* fix pspec check Signed-off-by:
-
Phuong Nguyen authored
* add manual axis filer to sharding_constraint impl Signed-off-by:
-
Kate Cheng authored
* Remove if-else and torch.tensor to meet cudagraph requirement Signed-off-by:
Kate Cheng <yunhsuanc@nvidia.com> * Add is_cg_capturable flag to guard the if-else statement Signed-off-by:
-
Paweł Gadziński authored
* code drop Signed-off-by:
-
Xin Yao authored
* enable PDL for blockwise qunatization kernels Signed-off-by:
-
jberchtold-nvidia authored
* Add L2_jax_distributed_unittest Signed-off-by:
-
- 12 Aug, 2025 4 commits
-
-
Jan Bielak authored
* Compute amax in normalization forward in current scaling in untuned kernels Signed-off-by:
-
jberchtold-nvidia authored
* Support setting collection name for quantizer set Flax variables in TransformerEngineBase flax module Signed-off-by:
-
Tim Moon authored
Fix bug when deducing dtype in linear functional API Signed-off-by: -
Tim Moon authored
* Update amax pointers when reallocating amax history in fusible ops Signed-off-by:
-
- 11 Aug, 2025 1 commit
-
-
Kirthi Shankar Sivamani authored
* Fix high-precision dtype for MXFP8 AG Signed-off-by:
-
- 09 Aug, 2025 1 commit
-
-
Daniel Stokes authored
* fix: Add stream synchronization before destroying MPI communicator (#1979) Signed-off-by:
djns99 <40156487+djns99@users.noreply.github.com> * feat: Implement column-wise userbuffer overlap for comm+GEMM operations Add support for overlapping column-wise allgather communication with GEMM operations to improve training performance: * **Core infrastructure changes:** - Update bulk_overlap_columnwise_ag() to accept explicit stream parameter - Modify userbuffers send/recv loops to use rank-ordered iteration - Add userbuffers_send_all/recv_all function declarations * **Python integration:** - Add bulk_overlap_ag_with_external_gemm() C++ extension function - Expose new overlap function via pybind11 bindings - Update overlap method configurations to include more ring_exchange ops * **LayerNorm MLP optimization:** - Enable column-wise quantization for FC2 gradient output - Implement overlap of allgather communication with FC2 DGRAD GEMM - Use fill_userbuffers_buffer_for_all_gather for efficient buffering This optimization allows overlapping communication and computation phases more effectively, reducing training wall-clock time by hiding allgather latency behind GEMM execution. Signed-off-by:
-
- 08 Aug, 2025 2 commits
-
-
Phuong Nguyen authored
* enabled TE GEMM for all recipes Signed-off-by:
-
Paweł Gadziński authored
* turn on userbuffers for layers without debug Signed-off-by:
-
