tests/test_tokenization_auto.py · ffecfea9495d4aa788e1c05d0612a40bc4b460fc · chenpangpang / transformers

fix `tokenizer_class_from_name` for models with `-` in the name (#13251) · 40d60e15

Stas Bekman authored Aug 26, 2021



* fix tokenizer_class_from_name

* Update src/transformers/models/auto/tokenization_auto.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* add test
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

40d60e15

test_tokenization_auto.py 8.21 KB

Replace test_tokenization_auto.py