tests/test_tokenization_bert.py · 65a89a89768f5922e51cdc7d49990d731e3f2c03 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

Fix BasicTokenizer to respect `never_split` parameters (#2557) · 65a89a89
Mark Neumann authored Jan 17, 2020
```
* add failing test

* fix call to _run_split_on_punc

* format with black
```
65a89a89

test_tokenization_bert.py 6.15 KB