tests/test_tokenization_common.py · e008d520bb9df64e7c2f644fd558301fd4276d51 · chenpangpang / transformers

Thomas Wolf authored Jun 24, 2020

* update tests for fast tokenizers + fix small bug in saving/loading

* better tests on serialization

* fixing serialization

* comment cleanup

7ac91107

test_tokenization_common.py 66.6 KB