src/transformers/tokenization_utils_base.py · dd52804f5fce0a568ffbb3dc7fd088d2de0a0e56 · chenpangpang / transformers

Sylvain Gugger authored Nov 17, 2020



* Remove old deprecated arguments
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>

* Remove needless imports

* Fix tests
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>

dd52804f

tokenization_utils_base.py 150 KB

Replace tokenization_utils_base.py