tests/test_tokenization_common.py · 9f8bfe703c4e8b88f462b23ff4968b1385bd955b · chenpangpang / transformers

Fix #16660 (tokenizers setters of ids of special tokens) (#16661) · 9f8bfe70

davidleonfdez authored Apr 13, 2022

* Fix setters of *_token_id properties of SpecialTokensMixin

* Test setters of common tokens ids

* Move to a separate test checks of setters of tokens ids

* Add independent test for ByT5

* Add Canine test

* Test speech to text

9f8bfe70

test_tokenization_common.py 199 KB

Replace test_tokenization_common.py