tests/tokenization/test_tokenization_utils.py · 3d7c2f9dea45338b7ebcd459b452e2fad7abfa1f · chenpangpang / transformers

#32184 save total_vocab_size (#32240) · 3d7c2f9d

Ita Zaporozhets authored Aug 05, 2024

* save total_vocab_size = vocab_size + user added tokens to speed up operation

* updating length when added_tokens_decoder is set

* add test len(tokenizer)

3d7c2f9d

test_tokenization_utils.py 13.4 KB

Replace test_tokenization_utils.py