tests/test_tokenization_common.py · d1ba56d8d8a4da183bfe7f0b30e0ec1220ebf6ea · chenpangpang / transformers

Improve tokenizer tests (#13594) · 66ea7391

Li-Huai (Allan) Lin authored Dec 03, 2021

* Use new method to acquire tokenizers

* Resolve TODOs.

* Style

* Fix

* Enable do_lower_case in test_tokenize_special_tokens

* Apply suggestion from code review

* Fix mask token handling

* Revert "Fix mask token handling"

This reverts commit daaa3f5291b1f71e5bc3604ca281c000000c4648.

* Fix FNet mask token tokenization

* Complete everything

* Apply suggestions from code review

66ea7391

test_tokenization_common.py 185 KB

Replace test_tokenization_common.py