tests/test_tokenization_bert.py · 21ed3a6b993eba06e7f4cf7720f4a07cc8a0d4c2 · chenpangpang / transformers

Reintroduce clean_text on BertTokenizer call which was removed by mistake in #4723 (#5749) · 21ed3a6b

Funtowicz Morgan authored Oct 09, 2020



* Reintroduce clean_text call which was removed by mistake in #4723
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

* Added unittest for clean_text parameter on Bert tokenizer.
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

* Better unittest name.
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

* Adapt unittest to use untrained tokenizer.
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

* Code quality + update test
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>

21ed3a6b

test_tokenization_bert.py 8.86 KB

Replace test_tokenization_bert.py