Unverified Commit d6ec54ba authored by Michael Benayoun's avatar Michael Benayoun Committed by GitHub
Browse files

solved coefficient issue for the TF version of gelu_fast (#11514)


Co-authored-by: default avatarMichael Benayoun <michael@huggingface.co>
parent ad1f7bef
......@@ -57,8 +57,8 @@ def mish(x):
def gelu_fast(x):
x = tf.convert_to_tensor(x)
coeff1 = tf.cast(0.7978845608, x.dtype)
coeff2 = tf.cast(0.044715, x.dtype)
coeff1 = tf.cast(0.044715, x.dtype)
coeff2 = tf.cast(0.7978845608, x.dtype)
return 0.5 * x * (1.0 + tf.tanh(x * coeff2 * (1.0 + coeff1 * x * x)))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment