tests/test_tokenization_clip.py · e93763d420910003656abe2f951ce5bd91ac2f08 · chenpangpang / transformers

fix CLIP fast tokenizer and change some properties of the slow version (#15067) · e93763d4

SaulLu authored Feb 18, 2022



Very big changes concerning the tokenizer fast of CLIP which did not correspond to the tokenizer slow of CLIP
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

e93763d4

test_tokenization_clip.py 8.49 KB

Replace test_tokenization_clip.py