tests/test_tokenization_auto.py · adb70eda4de20a4346bb3d085ffb959e7e588cb4 · chenpangpang / transformers

AutoTokenizer: infer the class from the tokenizer config if possible (#12208) · adb70eda

Sylvain Gugger authored Jun 17, 2021



* AutoTokenizer: infer the class from the tokenizer config if possible

* Add tests

* Update src/transformers/models/auto/tokenization_auto.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

adb70eda

test_tokenization_auto.py 7.44 KB

Replace test_tokenization_auto.py