• Arthur's avatar
    [`TokenizerFast`] `can_save_slow_tokenizer` as a property for when... · 3b39b906
    Arthur authored
    [`TokenizerFast`] `can_save_slow_tokenizer` as a property for when `vocab_file`'s folder was removed (#25626)
    
    * pad token should be None by default
    
    * fix tests
    
    * nits
    
    * check if isfile vocabfile
    
    * add warning if sp model folder was deleted
    
    * save SPM when missing folder for sloz
    
    * update the ` can_save_slow_tokenizer`  to be a property
    
    * first batch
    
    * second batch
    
    * missing one
    3b39b906
tokenization_bertweet.py 26.8 KB