tests/models/gpt2/test_tokenization_gpt2.py · 9e7f84a5563bd1198ff1e8858691713ef2b5493f · chenpangpang / transformers

[OPT] Adds `GPT2TokenizerFast` to the list of tokenizer to use for OPT. (#20823) · 9e7f84a5

Arthur authored Feb 07, 2023

* Add ("opt", ("GPT2Tokenizer", "GPT2TokenizerFast" if is_tokenizers_available() else None)),

* skip failing test

* Add ("opt", ("GPT2Tokenizer", "GPT2TokenizerFast" if is_tokenizers_available() else None)),

* skip failing test

9e7f84a5

test_tokenization_gpt2.py 13.1 KB

Replace test_tokenization_gpt2.py