src/transformers/tokenization_utils_base.py · 9017ba4ca40d10dc38940545a7b3ac76d89ba8e8 · chenpangpang / transformers

Sylvain Gugger authored Sep 16, 2022



* Fix tokenizer load from one file

* Add a test

* Style
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>

9017ba4c

tokenization_utils_base.py 173 KB