tests/test_tokenization_auto.py · 83bfdbdd7511360f944fc5ac5609d7f894a69041 · chenpangpang / transformers

"examples/vscode:/vscode.git/clone" did not exist on "2d42915abe19f1c36b90424f874f348357c2060f"

AutoTokenizer: infer the class from the tokenizer config if possible (#12208) · adb70eda

Sylvain Gugger authored Jun 17, 2021



* AutoTokenizer: infer the class from the tokenizer config if possible

* Add tests

* Update src/transformers/models/auto/tokenization_auto.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

adb70eda

test_tokenization_auto.py 7.44 KB

Replace test_tokenization_auto.py