• Arthur's avatar
    [`GemmaConverter`] use user_defined_symbols (#29473) · 2f9a3edb
    Arthur authored
    * use user_defined_symbols
    
    * fixup
    
    * nit
    
    * add a very robust test
    
    * make sure all models are tested with the `pretrained_tokenizer_to_test`
    
    * should we make sure we test all of them?
    
    * merge
    
    * remove the id
    
    * fix test
    
    * update
    
    * ousies
    
    * oups
    
    * fixup
    
    * fix copies check
    
    * remove `pretrained_tokenizer_to_test`
    2f9a3edb
test_tokenization_common.py 217 KB