Only use F.gelu for torch >=1.4.0 (#2955)

* Only use F.gelu for torch >=1.4.0 * Use F.gelu for newer torch

Only use F.gelu for torch >=1.4.0 (#2955)
* Only use F.gelu for torch >=1.4.0 * Use F.gelu for newer torch
b5b3445c · Sam Shleifer · GitHub · fc38d4c8 · b5b3445c
Unverified Commit b5b3445c authored Feb 21, 2020 by Sam Shleifer Committed by GitHub Feb 21, 2020
Hide whitespace changes
Inline Side-by-side

Showing with 4 additions and 1 deletion

src/transformers/activations.py src/transformers/activations.py +4 -1

No files found.
--- a/src/transformers/activations.py
+++ b/src/transformers/activations.py
@@ -18,7 +18,10 @@ def _gelu_python(x):
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))


-gelu = getattr(F, "gelu", _gelu_python)
+if torch.__version__ < "1.4.0":
+    gelu = _gelu_python
+else:
+    gelu = F.gelu


 def gelu_new(x):