• Sylvain Gugger's avatar
    Authorize last version of tokenizer (#9799) · 21b3922e
    Sylvain Gugger authored
    
    
    * Authorize last version of tokenizer
    
    * Update version table
    
    * Fix conversion of spm tokenizers and fix some hub links
    
    * Bump tokenizers version to 0.10.1rc1
    
    * Add script to check tokenizers conversion with XNLI
    
    * Add some more mask_token lstrip support
    
    * Must modify mask_token in slow tokenizers too
    
    * Keep using the old method for Pegasus
    
    * add missing import
    Co-authored-by: default avatarAnthony MOI <m.anthony.moi@gmail.com>
    21b3922e
setup.py 11.1 KB