Skip to content

GitLab

"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "defdcd286280ec6bddd19c81a308d35de67c3efd"
Unverified Commit d6ec54ba authored by Michael Benayoun's avatar Michael Benayoun Committed by GitHub
Browse files

solved coefficient issue for the TF version of gelu_fast (#11514) 


Co-authored-by: default avatarMichael Benayoun <michael@huggingface.co>
parent ad1f7bef
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
src/transformers/activations_tf.py
View file @ d6ec54ba
...@@ -57,8 +57,8 @@ def mish(x): ...@@ -57,8 +57,8 @@ def mish(x):
def gelu_fast(x): def gelu_fast(x):
x = tf.convert_to_tensor(x) x = tf.convert_to_tensor(x)
coeff1 = tf.cast(0.7978845608, x.dtype) coeff1 = tf.cast(0.044715, x.dtype)
coeff2 = tf.cast(0.044715, x.dtype) coeff2 = tf.cast(0.7978845608, x.dtype)
return 0.5 * x * (1.0 + tf.tanh(x * coeff2 * (1.0 + coeff1 * x * x))) return 0.5 * x * (1.0 + tf.tanh(x * coeff2 * (1.0 + coeff1 * x * x)))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment