Unverified Commit f9257843 authored by Yih-Dar's avatar Yih-Dar Committed by GitHub
Browse files

Fix incorrect model<->tokenizer mapping in tokenization testing (#19872)



* Fix model-tokenizer mapping
Co-authored-by: default avatarydshieh <ydshieh@users.noreply.github.com>
parent eedaba68
......@@ -120,9 +120,12 @@ def merge_model_tokenizer_mappings(
tokenizer = tokenizer_mapping[configuration][0]
tokenizer_fast = tokenizer_mapping[configuration][1]
model_tokenizer_mapping.update({tokenizer: (configuration, model)})
if tokenizer is not None:
if configuration.__name__.startswith(tokenizer.__name__.replace("Tokenizer", "")):
model_tokenizer_mapping.update({tokenizer: (configuration, model)})
if tokenizer_fast is not None:
model_tokenizer_mapping.update({tokenizer_fast: (configuration, model)})
if configuration.__name__.startswith(tokenizer_fast.__name__.replace("TokenizerFast", "")):
model_tokenizer_mapping.update({tokenizer_fast: (configuration, model)})
return model_tokenizer_mapping
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment