Skip to content

GitLab

"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "7e17f09fb5da60f533ad49bc4aace62e15755d2f"
Unverified Commit cc2366bb authored by Masatoshi Suzuki's avatar Masatoshi Suzuki Committed by GitHub
Browse files

Ignore not initialized NO_CONFIG_TOKENIZERs (#10936)

parent ddea8771
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
src/transformers/models/auto/tokenization_auto.py
View file @ cc2366bb
...@@ -288,7 +288,7 @@ def tokenizer_class_from_name(class_name: str): ...@@ -288,7 +288,7 @@ def tokenizer_class_from_name(class_name: str):
all_tokenizer_classes = ( all_tokenizer_classes = (
[v[0] for v in TOKENIZER_MAPPING.values() if v[0] is not None] [v[0] for v in TOKENIZER_MAPPING.values() if v[0] is not None]
+ [v[1] for v in TOKENIZER_MAPPING.values() if v[1] is not None] + [v[1] for v in TOKENIZER_MAPPING.values() if v[1] is not None]
+ NO_CONFIG_TOKENIZER + [v for v in NO_CONFIG_TOKENIZER if v is not None]
) )
for c in all_tokenizer_classes: for c in all_tokenizer_classes:
if c.__name__ == class_name: if c.__name__ == class_name:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment