tests/test_tokenization_fast.py · 786ced36390e9d1c4c5c32a81d7f00c36913aa2a · chenpangpang / transformers

Add versioning system to fast tokenizer files (#12713) · 786ced36

Sylvain Gugger authored Jul 21, 2021



* Add versioning system to fast tokenizer files

* Deal with offline mode

* Use staging env in tests

* Style

* Apply suggestions from code review
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Style
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

786ced36

test_tokenization_fast.py 7.61 KB

Replace test_tokenization_fast.py