tests/models/wav2vec2/test_tokenization_wav2vec2.py · 15cd68713d8d027e1033906bf39e999a24b5b5dd · chenpangpang / transformers

Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode` (#29311) · 15cd6871

Minsub Lee (Matt) authored Apr 02, 2024

* Fix skip_special_tokens process for Wav2Vec2CTCTokenizer._decode

* Fix skip_special_tokens for Wav2Vec2CTCTokenizer._decode

* Exclude pad_token filtering since it is used as CTC-blank token

* Add small test for skip_special_tokens

* Update decoding test for added new token

15cd6871

test_tokenization_wav2vec2.py 39.9 KB

Replace test_tokenization_wav2vec2.py