tests/models/whisper/test_tokenization_whisper.py · 7adce8b53241bd0f537c5135541fbc166dbe9be8 · chenpangpang / transformers

fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for... · 7adce8b5

Connor Henderson authored May 31, 2023

fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)

* add ' ' replacement for add_prefix_space

* add fast tokenizer test

7adce8b5

test_tokenization_whisper.py 19.2 KB

Replace test_tokenization_whisper.py