tests/models/llama/test_tokenization_llama.py · a2a7f71604ebc930ed6cbc594eb0594811e24bc8 · chenpangpang / transformers

[ `TokenizationLlama`] fix the way we convert tokens to strings to keep... · a2a7f716

Arthur authored Mar 28, 2024

[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix (#29453)

* nit

* update test and fix test

* fixup

a2a7f716

test_tokenization_llama.py 34.8 KB

Replace test_tokenization_llama.py