tests/tokenization/test_tokenization_fast.py · f9cc333805c47665c6afee8b5867931e54abe0c6 · chenpangpang / transformers

[ `PreTrainedTokenizerFast`] Keep properties from fast tokenizer (#25053) · f9cc3338

Arthur authored Jul 25, 2023

* draft solution

* use `setdefault`

* nits

* add tests and fix truncation issue

* fix test

* test passes locally

* quality

* updates

* update tsets

f9cc3338

test_tokenization_fast.py 11.4 KB

Replace test_tokenization_fast.py