tests/test_tokenization_common.py · 6e87010060ddfb334293df4228123ed7a4ca6ad7 · chenpangpang / transformers

Preserve `list` type of `additional_special_tokens` in `special_token_map` (#12759) · 6e870100

SaulLu authored Jul 16, 2021



* preserve type of `additional_special_tokens` in `special_token_map`

* format

* Update src/transformers/tokenization_utils_base.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

6e870100

test_tokenization_common.py 168 KB

Replace test_tokenization_common.py