tests/test_tokenization_common.py · f6b44e61903964aee07bdad7056a1ad9d0006e6f · chenpangpang / transformers

Tokenizers should be framework agnostic (#8599) · 3095ee9d

Lysandre Debut authored Nov 17, 2020



* Tokenizers should be framework agnostic

* Run the slow tests

* Not testing

* Fix documentation

* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

3095ee9d

test_tokenization_common.py 139 KB

Replace test_tokenization_common.py