- 09 Oct, 2024 2 commits
-
-
Charlene Yang authored
* improve get_attention_backend logic Signed-off-by:
Charlene Yang <8636796+cyanguwa@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * polish logic and wording Signed-off-by:
pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
-
Charlene Yang authored
* add extra_state change description for different TE versions Signed-off-by:
-
- 08 Oct, 2024 1 commit
-
-
Charlene Yang authored
* add qkv descales to FA3 Signed-off-by:
-
- 07 Oct, 2024 1 commit
-
-
Xiaowei Ren authored
* change API for hierarchical CP Signed-off-by:
Xiaowei Ren <xren@nvidia.com> * move fp8 code before qkv reshape Signed-off-by:
-
- 03 Oct, 2024 1 commit
-
-
Charlene Yang authored
move block_table arg to varlen_func section Signed-off-by:
-
- 27 Sep, 2024 1 commit
-
-
Paweł Gadziński authored
* Docs fixes Signed-off-by:
Pawel Gadzinski <pgadzinski@nvidia.com> * docs fix Signed-off-by:
-
- 19 Sep, 2024 1 commit
-
-
Xin Yao authored
* relax contiguous check for flash attention Signed-off-by:
Xin Yao <xiny@nvidia.com> * force contiguous for cp Signed-off-by:
-
- 18 Sep, 2024 1 commit
-
-
Sudhakar Singh authored
* make rotary_base arg Signed-off-by:
Sudhakar Singh <sudhakars@nvidia.com> * rotary base can be a float Co-authored-by:
Tim Moon <4406448+timmoon10@users.noreply.github.com> Signed-off-by:
-
- 09 Sep, 2024 1 commit
-
-
Xiaowei Ren authored
* clean code for CP function args Signed-off-by:
-
- 05 Sep, 2024 2 commits
-
-
Selvaraj Anandaraj authored
* Added offloading support FP8 attention Signed-off-by:
Selvaraj Anandaraj <selvaraja@login-eos02.eos.clusters.nvidia.com> * Update transformer_engine/pytorch/attention.py Co-authored-by:
Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by:
Selvaraj Anandaraj <anandaraj@wisc.edu> * Fix Signed-off-by:
-
Xin Yao authored
* fp8 mha with rope Signed-off-by:
-
- 30 Aug, 2024 2 commits
-
-
Xiaowei Ren authored
* fix qkv_dtype of FP8+CP Signed-off-by:
-
Charlene Yang authored
* fix FP8 logic when FA3 is not installed Signed-off-by:
-
- 23 Aug, 2024 1 commit
-
-
Charlene Yang authored
* WIP: add fa3 Signed-off-by:
-
- 21 Aug, 2024 2 commits
-
-
Charlene Yang authored
* add support for padding in UnfusedDPA Signed-off-by:
-
Xiaowei Ren authored
* add window_size to AttnFuncWithCP Signed-off-by:
Xiaowei Ren <xren@cs-cw-dfw-login-01.cm.cluster>
-
- 20 Aug, 2024 1 commit
-
-
hXl3s authored
feat(pytorch): Allow TransformerLayer and MultiheadAttention to accept sequence length parameters (#1066) * Added ability for seqlen for transformer and mha layer Signed-off-by:
Lukasz Pierscieniewski <lukaszp@nvidia.com> * Documentation for new parameters Signed-off-by:
Przemek Tredak <ptredak@nvidia.com> --------- Signed-off-by:
-
- 16 Aug, 2024 1 commit
-
-
Xiaowei Ren authored
* add window_size to AttnFuncWithCP Signed-off-by:
-
- 15 Aug, 2024 2 commits
-
-
Charlene Yang authored
fix typos regarding t in thd Signed-off-by: -
Marks101 authored
Signed-off-by:
Markus Schnoes <markus.schnoes@gmx.de> Co-authored-by:
-
- 13 Aug, 2024 1 commit
-
-
Charlene Yang authored
* merge k_channels and v_channels back to kv_channels and accept a tuple Signed-off-by:
-
- 06 Aug, 2024 3 commits
-
-
Charlene Yang authored
reduce the roundup of max_seqlen for THD to multiples of 64 Signed-off-by: -
Charlene Yang authored
* fix logging in attention Signed-off-by:
-
Charlene Yang authored
* add multi-latent attention for DPA Signed-off-by:
-
- 02 Aug, 2024 1 commit
-
-
Li Tao authored
fix an argument issue when flash_attn>=2.5.7 Signed-off-by:
Li Tao <lit@nvidia.com> Co-authored-by:
-
- 01 Aug, 2024 1 commit
-
-
Xiaowei Ren authored
* use 2hd layout Signed-off-by:
-
- 26 Jul, 2024 1 commit
-
-
Charlene Yang authored
fix tp_size for GQA Signed-off-by:
-
- 25 Jul, 2024 1 commit
-
-
李金梁 authored
This bug will cause bug [ERROR] failed (exitcode: -11) local_rank: 0 (pid: 1761020) of binary: ~/megatron/bin/python. That is because we miss the rng_states that is required in attention recompute (for dropout), but no hint is provided. It is very very very difficult to trace and cost me two weeks. ```python before the start of training step] datetime: 2024-07-22 18:26:45 [2024-07-22 18:27:00,941] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -11) local_rank: 0 (pid: 1761020) of binary: /home//miniconda3/envs/megatron/bin/python Traceback (most recent call last): File "/home//miniconda3/envs/megatron/bin/torchrun", line 33, in <module> sys.exit(load_entry_point('torch==2.2.1+cu121', 'console_scripts', 'torchrun')()) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home//miniconda3/envs/megatron/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ``` Signed-off-by:李金梁 <975761915@qq.com>
-
- 19 Jul, 2024 1 commit
-
-
Charlene Yang authored
* initialize output tensors to 0 for THD while waiting for cuDNN bug fix Signed-off-by:
-
- 10 Jul, 2024 1 commit
-
-
Charlene Yang authored
* add cuDNN swa Signed-off-by:
-
- 03 Jul, 2024 1 commit
-
-
Charlene Yang authored
* update to FE 1.5.1 and add bottom right causal Signed-off-by:
-
- 01 Jul, 2024 1 commit
-
-
Charlene Yang authored
* update FE to 1.5.2 Signed-off-by:
-
- 18 Jun, 2024 2 commits
-
-
Charlene Yang authored
* simplify offset tensors Signed-off-by:
-
Charlene Yang authored
fix tp_initialized error Signed-off-by:
-
- 15 Jun, 2024 1 commit
-
-
Charlene Yang authored
* subclass DPA with BaseModule and test with test_gpt_checkpointing Signed-off-by:
-
- 14 Jun, 2024 3 commits
-
-
Kirthi Shankar Sivamani authored
* Apply formatting Signed-off-by:
-
Kirthi Shankar Sivamani authored
* Initial config test Signed-off-by:
-
Charlene Yang authored
* add attention docs Signed-off-by:
-
- 13 Jun, 2024 1 commit
-
-
BoxiangW authored
* Add norm_factor arg into DotProductAttention Signed-off-by:
Boxiang Wang <boxiangw@nvidia.com> * Change kwarg name from `norm_factor` to `softmax_scale` Signed-off-by:
BoxiangW <45734921+BoxiangW@users.noreply.github.com> --------- Signed-off-by:
-
- 10 Jun, 2024 1 commit
-
-
Xiaowei Ren authored
* add seq_offsets_qkvo for cudnn thd Signed-off-by:
-
