Adding `Perceiver` to `AutoTokenizer`. (#14711)

5b004001 · Nicolas Patry · GitHub · 59d684fa · 5b004001
Unverified Commit 5b004001 authored Dec 10, 2021 by Nicolas Patry Committed by GitHub Dec 10, 2021
Hide whitespace changes
Inline Side-by-side

Showing with 7 additions and 0 deletions

src/transformers/models/auto/tokenization_auto.py src/transformers/models/auto/tokenization_auto.py +7 -0

No files found.
--- a/src/transformers/models/auto/tokenization_auto.py
+++ b/src/transformers/models/auto/tokenization_auto.py
@@ -223,6 +223,13 @@ else:
                    "CLIPTokenizerFast" if is_tokenizers_available() else None,
                ),
            ),
+            (
+                "perceiver",
+                (
+                    "PerceiverTokenizer",
+                    None,
+                ),
+            ),
        ]
    )