Skip to content

GitLab

Switch branch/tag
  1. 05 Aug, 2022 1 commit
    • Hubert Lu's avatar
      Enable FusedRMSNorm (#78) · c97ebfab
      Hubert Lu authored 
      

* FusedRMSNorm/"T5LayerNorm" based on FusedLayerNorm (#1274)

* FusedRMSNorm based on FusedLayerNorm

* refactor duplicated kernels

* delete comments

* delete comments

* cleanup

* cleanup

* cleanup, fixed clobbering forward_affine_mixed_dtypes

* fix pybind naming and add MixedFused test

* undo skipping

* check elementwise_affine

* Update tests/L0/run_fused_layer_norm/test_fused_layer_norm.py

Oof, nice catch, thanks
Co-authored-by: default avatarMasaki Kozuki <masaki.kozuki.2014@gmail.com>
Co-authored-by: default avatarMasaki Kozuki <masaki.kozuki.2014@gmail.com>

* fix and generate docs for FusedRMSNorm (#1285)

* [FusedRMSNorm doc] document where epsilon is added (#1295)

* [FusedRMSNorm doc] add epsilon to formula

* correct

* better wording

* Fix some bugs

* Optimize HostRMSNormGradient and HostApplyRMSNorm for AMD GPUs

* Fix NaN issues in FusedRMSNorm

* Update test_fused_layer_norm.py

* Skip test_fused_layer_norm.TestAutocastFusedRMSNorm on ROCm

* Use at::cuda::warp_size() instead of at::cuda::getCurrentDeviceProperties()->warpSize
Co-authored-by: default avatareqy <eddiey@nvidia.com>
Co-authored-by: default avatarMasaki Kozuki <masaki.kozuki.2014@gmail.com>
Co-authored-by: default avatarStas Bekman <stas00@users.noreply.github.com>
      c97ebfab
  3. 29 Jul, 2022 1 commit
  5. 15 Apr, 2022 4 commits
  7. 04 Feb, 2022 1 commit
  9. 25 Jan, 2022 1 commit
  11. 17 Nov, 2021 1 commit
  13. 08 Oct, 2021 1 commit
  15. 07 Oct, 2021 1 commit
  17. 02 Oct, 2021 1 commit
  19. 24 Sep, 2021 1 commit
  21. 04 Nov, 2020 1 commit
  23. 15 Jun, 2020 1 commit
  25. 26 May, 2020 1 commit
  27. 07 May, 2020 1 commit
  29. 06 Sep, 2019 1 commit
    • mcarilli's avatar
      Fix for #456 (#477) · 325f5a0b
      mcarilli authored 
      * Pushing for build tests

* Contrib files

* Removing deprecated checks
      325f5a0b
  31. 06 Aug, 2019 1 commit
    • ngimel's avatar
      Clean up layer norm tests (#418) · 3ef01fae
      ngimel authored 
      * Bug fix for non-affine layer-norm + add backward unit test

* clean up tests and add tests for a large batch
      3ef01fae
  33. 01 Aug, 2019 1 commit
  35. 26 Apr, 2019 1 commit
    • ptrblck's avatar
      Replace type().ScalarType() with scalar_type() (#272) · 855808f3
      ptrblck authored 
      * change .type().ScalarType() to .scalar_type() + at::ScalarType::X to at::kX

* revert scalar_type() to type() for AT_DISPATCH_FLOATING_TYPES_AND_HALF

* revert scalar_type() to type() in AT_DISPATCH_FLOATING_TYPES

* revert scalar_type() to type() for AT_DISPATCH_FLOATING_TYPES_AND_HALF in welford.cu

* revert scalar_type() to type() in layer_norm_cuda_kernel.cu

* revert at::kType  to at::ScalarType::Type

* use DISPATCH_FLOAT_AND_HALF to get rid of warnings

* add dispatch mechanisms for double+float and double+float+half
      855808f3
  37. 08 Apr, 2019 1 commit
  39. 21 Mar, 2019 2 commits
  41. 15 Mar, 2019 1 commit
  43. 12 Mar, 2019 1 commit
  45. 31 Oct, 2018 1 commit