Some `model_type`s cannot be in the mapping (#13259)

* Some tokenizers cannot be in the mapping * Style

Some `model_type`s cannot be in the mapping (#13259)
* Some tokenizers cannot be in the mapping * Style
5af8df5a · Lysandre Debut · GitHub · 68b69072 · 5af8df5a
Unverified Commit 5af8df5a authored Aug 25, 2021 by Lysandre Debut Committed by GitHub Aug 25, 2021
Hide whitespace changes
Inline Side-by-side

Showing with 3 additions and 1 deletion

src/transformers/tokenization_utils_base.py src/transformers/tokenization_utils_base.py +3 -1

No files found.
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1799,7 +1799,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
                            break

                if model_type is not None:
-                    config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES[model_type]
+                    config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES.get(
+                        model_type, (None, None)
+                    )
                    if config_tokenizer_class is None:
                        config_tokenizer_class = config_tokenizer_class_fast