tests/models/esm/test_tokenization_esm.py · d15781597a23a215306e4192ba22b7e949e12c13 · chenpangpang / transformers

Allow add_tokens for ESM (#28535) · d1578159

Matt authored Jan 19, 2024



* Allow non-special tokens to be added

* Add test, fix token adding code

* Revert changes to id_to_token and token_to_id

* Update the ESM tokenizer to be a bit more standardized

* Update src/transformers/models/esm/tokenization_esm.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

d1578159

test_tokenization_esm.py 4.84 KB

Replace test_tokenization_esm.py