tests/test_tokenization_xlm_roberta.py · e02ed0ee7e1b500010452b569087f4e6ddd1f800 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

XLMR tokenizer is fully picklable (#13577) · e02ed0ee
Benjamin Davidson authored Sep 16, 2021
```
* made tokenizer fully picklable

* remove whitespace

* added testcase
```
e02ed0ee

test_tokenization_xlm_roberta.py 10.7 KB