Skip to content

GitLab

"src/git@developer.sourcefind.cn:renzhc/diffusers_dcu.git" did not exist on "c7a39d38adb36059b31318b6841dda4e8f6ab172"
Commit 47921708 authored by Stas Bekman's avatar Stas Bekman Committed by hubertlu-tw
Browse files

[FusedRMSNorm doc] document where epsilon is added (#1295) 

* [FusedRMSNorm doc] add epsilon to formula

* correct

* better wording
parent fceec07d
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
apex/normalization/fused_layer_norm.py
View file @ 47921708
...@@ -310,6 +310,7 @@ class FusedRMSNorm(torch.nn.Module): ...@@ -310,6 +310,7 @@ class FusedRMSNorm(torch.nn.Module):
:attr:`normalized_shape`. :attr:`normalized_shape`.
:math:`\gamma` is a learnable affine transform parameter of :math:`\gamma` is a learnable affine transform parameter of
:attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``. :attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``.
`epsilon` is added to the mean-square, then the root of the sum is taken.
.. note:: .. note::
Unlike Batch Normalization and Instance Normalization, which applies Unlike Batch Normalization and Instance Normalization, which applies
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment