tests/test_tokenization_auto.py · 78807d86eb8791bdbe149adff62af11f778a9267 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

Fix AutoTokenizer when no fast tokenizer is available (#13336) · c4ecd234
Sylvain Gugger authored Aug 30, 2021
```
* Fix AutoTokenizer when a tokenizer has no fast version

* Add test
```
c4ecd234

test_tokenization_auto.py 8.46 KB