• SaulLu's avatar
    add a warning in `SpmConverter` for sentencepiece's model using the byte fallback feature (#16629) · 1025a9b7
    SaulLu authored
    * update proto sentencepiece model
    
    * Revert "update proto sentencepiece model"
    
    This reverts commit b07f671747fec35773d0b3d4788b8b15aefa0229.
    
    * add check
    
    * add test
    
    * Revert "Revert "update proto sentencepiece model""
    
    This reverts commit 46108257b8927b73627ec8f4f3eed53a95fc700d.
    
    * test for log level
    
    * test for log level 2
    
    * warning at the warning level
    
    * clean
    
    * format
    
    * add explanation in docstring
    1025a9b7