tests/test_tokenization_bert.py · e52d1258e010b88d3507a4f527c6201616c119ad · chenpangpang / transformers · GitLab

".circleci/vscode:/vscode.git/clone" did not exist on "092a786984d4e69812e4de538c7a40921d5d1281"

Find file Blame History Permalink

Fix BasicTokenizer to respect `never_split` parameters (#2557) · 65a89a89
Mark Neumann authored Jan 17, 2020
```
* add failing test

* fix call to _run_split_on_punc

* format with black
```
65a89a89

test_tokenization_bert.py 6.15 KB