Skip to content

GitLab

Unverified Commit 434d15da authored by Dan Hendrycks's avatar Dan Hendrycks Committed by GitHub
Browse files

Update activation function docstring

parent 5faf3866
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
pytorch_pretrained_bert/modeling.py
View file @ 434d15da
...@@ -114,6 +114,7 @@ def gelu(x): ...@@ -114,6 +114,7 @@ def gelu(x):
"""Implementation of the gelu activation function. """Implementation of the gelu activation function.
For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
Also see https://arxiv.org/abs/1606.08415
""" """
return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0))) return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment