- 02 Aug, 2024 1 commit
-
-
Przemyslaw Tredak authored
* Link attention docs to the main docs and fix errors reported by Sphinx Signed-off-by:
Przemek Tredak <ptredak@nvidia.com> * Lower the version of nbsphinx Signed-off-by:
-
- 01 Aug, 2024 2 commits
-
-
Xiaowei Ren authored
* use 2hd layout Signed-off-by:
Xiaowei Ren <xren@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change qkv_format check Signed-off-by:
pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by:
Charlene Yang <8636796+cyanguwa@users.noreply.github.com> Co-authored-by:
Xiaowei Ren <xren@cs-cw-dfw-login-01.cm.cluster>
-
Xin Yao authored
* fix workspaces and unfused bias in multi-stream cuBLAS * Expose num_streams via pybind * Fix C-compatibility * rm importing packaging in test_fused_attn.py --------- Signed-off-by:
Xin Yao <xiny@nvidia.com> Co-authored-by:
Phuong Nguyen <phuonguyen@nvidia.com>
-
- 31 Jul, 2024 2 commits
-
-
Przemyslaw Tredak authored
* Added tests for silu/relu/swiglu/reglu Signed-off-by:
-
Przemyslaw Tredak authored
* Ensure that the inputs to custom calls are contiguous Signed-off-by:
Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by:
-
- 30 Jul, 2024 2 commits
-
-
Selvaraj Anandaraj authored
* Load balanced offloading algorithm Signed-off-by:
Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by:
-
Shijie authored
add deterministic option Signed-off-by:
Shijie Wang <jaywan@nvidia.com> Co-authored-by:
-
- 26 Jul, 2024 1 commit
-
-
Charlene Yang authored
fix tp_size for GQA Signed-off-by:
-
- 25 Jul, 2024 5 commits
-
-
李金梁 authored
This bug will cause bug [ERROR] failed (exitcode: -11) local_rank: 0 (pid: 1761020) of binary: ~/megatron/bin/python. That is because we miss the rng_states that is required in attention recompute (for dropout), but no hint is provided. It is very very very difficult to trace and cost me two weeks. ```python before the start of training step] datetime: 2024-07-22 18:26:45 [2024-07-22 18:27:00,941] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -11) local_rank: 0 (pid: 1761020) of binary: /home//miniconda3/envs/megatron/bin/python Traceback (most recent call last): File "/home//miniconda3/envs/megatron/bin/torchrun", line 33, in <module> sys.exit(load_entry_point('torch==2.2.1+cu121', 'console_scripts', 'torchrun')()) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ``` Signed-off-by:李金梁 <975761915@qq.com>
-
Kirthi Shankar Sivamani authored
* Fixes for wheels Signed-off-by:
-
Kirthi Shankar Sivamani authored
* Specify python version Signed-off-by:
-
Tim Moon authored
Fix linter warnings Signed-off-by:Tim Moon <tmoon@nvidia.com>
-
Tim Moon authored
* Fix build error with Paddle >2.6.1 Signed-off-by:
-
- 24 Jul, 2024 2 commits
-
-
Tian Zheng authored
* i Signed-off-by:
Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * . Signed-off-by:
-
Tim Moon authored
* Set minimum CMake version to 3.21 Stop linking to nvtx. Signed-off-by:
-
- 23 Jul, 2024 2 commits
-
-
Alp Dener authored
[PyTorch] Fixing hang in `initialize_ub()` for multi-node runs after PR901 removal of MPI-dependence (#986) * Re-implementing PR901 (removing MPI-dependence in Userbuffers) with multi-node fixes * passing data-parallel rank/size info from torch.distributed to userbuffers Signed-off-by:
Alp Dener <adener@nvidia.com> * multi-node example working with UB_SKIPMC=1 but not with multicast Signed-off-by:
-
Selvaraj Anandaraj authored
* removed unwanted memcpyDtoD/fixed weight parametrisation Signed-off-by:
-
- 22 Jul, 2024 1 commit
-
-
Selvaraj Anandaraj authored
* Fixed convergence issues Signed-off-by:
-
- 21 Jul, 2024 1 commit
-
-
Tim Moon authored
* Update sequential container constructor to handle modules in plain dicts Signed-off-by:
-
- 19 Jul, 2024 1 commit
-
-
Charlene Yang authored
* initialize output tensors to 0 for THD while waiting for cuDNN bug fix Signed-off-by:
-
- 18 Jul, 2024 2 commits
-
-
Reese Wang authored
Update nvtx header Signed-off-by:Reese Wang <rewang@nvidia.com>
-
Alp Dener authored
* FP8 type switch macro now wraps only the FP8 kernel to avoid invalid type errors Signed-off-by:
-
- 17 Jul, 2024 4 commits
-
-
Frank Lin authored
fix 261 compile Signed-off-by:
Frank Lin (Engrg-Hardware 1) <eee4017@gmail.com> Co-authored-by:
Frank Lin (Engrg-Hardware 1) <fralin@nvidia.com> Co-authored-by:
-
vasunvidia authored
* DGRAD_RS UB overlap Bug fixes Signed-off-by:
Vasudevan Rengasamy <vrengasamy@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by:
-
Tim Moon authored
* Add option to pass kwargs to CUDA graph module Signed-off-by:
-
Reese Wang authored
* Add enabled() to BasePrimitive * Add layernorm/rmsnorm fallback * Add cast_fp8 fallback * Add transpose/cast_transpose XLA fall back * Act_lu fallback * Add transpose fallback * Add softmax fallback * Unify the use of _cast_fp8 * Add tests for NVTE_JAX_CUSTOM_CALLS_RE --------- Signed-off-by:
Phuong Nguyen <36155692+phu0ngng@users.noreply.github.com>
-
- 10 Jul, 2024 2 commits
-
-
Charlene Yang authored
* add cuDNN swa Signed-off-by:
-
Tim Moon authored
Reduce CUDA driver API calls when choosing transpose kernels Signed-off-by:
-
- 09 Jul, 2024 2 commits
-
-
Tim Moon authored
* Add basic infrastructure for Sequential module Signed-off-by:
-
Phuong Nguyen authored
* removed singleton wrapper Signed-off-by:
-
- 08 Jul, 2024 4 commits
-
-
Kirthi Shankar Sivamani authored
Build for python < 3.8 Signed-off-by: -
Phuong Nguyen authored
* add parallel build without pyproject Signed-off-by:
-
Min Yu authored
[Paddle] Fix forward and backward logic of te.Linear(parallel_mode='column') to adapt DiT of PaddleMIX (#963) [Paddle] Fix forward and backward of Linear(parallel_mode='column') When te.Linear(parallel_mode='column') is not used in pairs with te.Linear(parallel_mode='row'), the output should to be all-gathered when forward and reduce-scattered when backward. Signed-off-by:
minyu <minyu@nvidia.com> Co-authored-by:
-
Xin Yao authored
* remove implicit padding and unpadding Signed-off-by:
-
- 05 Jul, 2024 1 commit
-
-
Zhenhuan Liu authored
Fix size mismatch error in fp8 transpose. Signed-off-by:Dennis Liu <denliu@nvidia.com>
-
- 03 Jul, 2024 3 commits
-
-
Charlene Yang authored
* update to FE 1.5.1 and add bottom right causal Signed-off-by:
-
Alp Dener authored
* removed libcuda.so link at compile time for TE/PyTorch extension Signed-off-by:
-
Reese Wang authored
* Integrate experimental ragged offset Signed-off-by:
-
- 02 Jul, 2024 2 commits
-
-
Tim Moon authored
Fix typo when selecting tuned RMSNorm kernels Signed-off-by: -
Frank Lin authored
* NVTE_OVERRIDE_MAX_SEQ_LEN Signed-off-by:
-
