tests/test_tokenization_common.py · 9017ba4ca40d10dc38940545a7b3ac76d89ba8e8 · chenpangpang / transformers

Sylvain Gugger authored Sep 16, 2022



* Fix tokenizer load from one file

* Add a test

* Style
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>

9017ba4c

test_tokenization_common.py 203 KB