tests/models/llama/test_tokenization_llama.py · 5b5ee235f3239413e9614bd02032b1a203dab710 · chenpangpang / transformers

[`LlamaTokenizer`] `tokenize` nits. (#25793) · 5b5ee235

Arthur authored Aug 29, 2023



* return when length is zero

* Add tests
Co-authored-by: Avnish Narayan <38871737avnishn@users.noreply.github.com>

* Co-authored-by: avnishn
<38871737+avnishn@users.noreply.github.com>

* codeLlama doc should not be on Main

* update test

---------
Co-authored-by: Avnish Narayan <38871737avnishn@users.noreply.github.com>

5b5ee235

test_tokenization_llama.py 30.1 KB

Replace test_tokenization_llama.py