Fixed the default number of attention heads in Reformer Configuration (#6973)

f7e80721 · tznurmin · GitHub · e20d8895 · f7e80721
Unverified Commit f7e80721 authored Sep 07, 2020 by tznurmin Committed by GitHub Sep 07, 2020
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

src/transformers/configuration_reformer.py src/transformers/configuration_reformer.py +1 -1

No files found.
--- a/src/transformers/configuration_reformer.py
+++ b/src/transformers/configuration_reformer.py
@@ -160,7 +160,7 @@ class ReformerConfig(PretrainedConfig):
        lsh_num_chunks_before=1,
        lsh_num_chunks_after=0,
        max_position_embeddings=4096,
-        num_attention_heads=2,
+        num_attention_heads=12,
        num_buckets=None,
        num_hashes=1,
        pad_token_id=0,