GitLab

* update ngc link and dockerhub container tag

* update

* update

* update

* Update README.md
Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com>

[transformer] Warn only when `gradient_accumulation_fusion` is `True` and `fused_weight_gradient_mlp_cuda` is missing (#1317)

* initial issue_template -- bug

* Apply suggestions from code review
Co-authored-by: eqy <eqy@cs.washington.edu>
Co-authored-by: eqy <eqy@cs.washington.edu>

* Move forward cudnn-frontend

* update throw_if to adapt cudnn frontend

Leave bottleneck masks as bool

* extend api to allow forced memory zeroing (empty() does not do it)

* typo fix

* ctx change

* move zeroing flag to ctx

* update test
Co-authored-by: mchochowski <mchochowski@nvidia.com>
Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com>

This reverts commit adbe075a.

This reverts commit 74e04667.

* update build_model to support enc&dec model

* fix typo: cur_sargs -> cur_args

* enc&dec path: correctly update pre/post process

* fuse grad accumulation w/ weight grad
Co-authored-by: Sangkug Lym <slym@nvidia.com>

* fp32 training path

* not using *args, **kwargs

* backward: moved the tensor dimension cnversion
Co-authored-by: Sangkug Lym <slym@nvidia.com>

* move files to csrc/megatron

* fix fp32 path

* fix typo

* add  to  in order to select the correct custom extension

* fix typo

* comment on import guard

* update test: enable gradient_accumulation_fusion

* 86

* remove redundant call of `test_column_parallel_linear`
Co-authored-by: Sangkug Lym <slym@nvidia.com>

Change data type for virtual tensors to float

* [FusedRMSNorm doc] add epsilon to formula

* correct

* better wording

* FusedRMSNorm based on FusedLayerNorm

* refactor duplicated kernels

* delete comments

* delete comments

* cleanup

* cleanup

* cleanup, fixed clobbering forward_affine_mixed_dtypes

* fix pybind naming and add MixedFused test

* undo skipping

* check elementwise_affine

* Update tests/L0/run_fused_layer_norm/test_fused_layer_norm.py

Oof, nice catch, thanks
Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>
Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>