tests/models/llama/test_tokenization_llama.py · 6df9179c1c5b19674ce0a4b82d311ef833c02447 · chenpangpang / transformers

[`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606) · 9ad815e4

Arthur authored Oct 06, 2023

* make sure eos and bos are properly handled for fast tokenizer

* fix code llama as well

* nits

* fix the conversion script as well

* fix failing test

9ad815e4

test_tokenization_llama.py 33.1 KB

Replace test_tokenization_llama.py