tests/test_tokenization_common.py · b54ef78d0c30045bb3f9ecc8b178eab0dfdbeaec · chenpangpang / transformers

Testing that batch_encode_plus is the same as encode_plus (#2973) · 21d8b6a3

Lysandre Debut authored Feb 24, 2020

* Testing that encode_plus and batch_encode_plus behave the same way

Spoiler alert: they don't

* Testing rest of arguments in batch_encode_plus

* Test tensor return in batch_encode_plus

* Addressing Sam's comments

* flake8

* Simplified with `num_added_tokens`

21d8b6a3

test_tokenization_common.py 28.6 KB

Replace test_tokenization_common.py