tests/test_tokenization_xlm_roberta.py · f80775df2b418716acce76d56826ed59183869b6 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

XLMR tokenizer is fully picklable (#13577) · e02ed0ee
Benjamin Davidson authored Sep 16, 2021
```
* made tokenizer fully picklable

* remove whitespace

* added testcase
```
e02ed0ee

test_tokenization_xlm_roberta.py 10.7 KB