Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "edba935d6708a7bc7f9589007da5de1aee8138cc"
Unverified Commit 5af8df5a authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Some `model_type`s cannot be in the mapping (#13259) 

* Some tokenizers cannot be in the mapping

* Style
parent 68b69072
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 1 deletion
src/transformers/tokenization_utils_base.py
View file @ 5af8df5a
...@@ -1799,7 +1799,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin): ...@@ -1799,7 +1799,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
break break
if model_type is not None: if model_type is not None:
config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES[model_type] config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES.get(
model_type, (None, None)
)
if config_tokenizer_class is None: if config_tokenizer_class is None:
config_tokenizer_class = config_tokenizer_class_fast config_tokenizer_class = config_tokenizer_class_fast
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment