tests/test_tokenization_fast.py · e67676424191e5935362e5fe7e04b5c317d706a9 · chenpangpang / transformers

Override build_inputs_with_special_tokens for fast tokenizers (#2912) · e6767642

Funtowicz Morgan authored Feb 19, 2020



* Override build_inputs_with_special_tokens for fast impl + unittest.
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Quality + format.
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

e6767642

test_tokenization_fast.py 19.7 KB

Replace test_tokenization_fast.py