1. 04 Feb, 2021 1 commit
    • Sylvain Gugger's avatar
      Authorize last version of tokenizer (#9799) · 21b3922e
      Sylvain Gugger authored
      
      
      * Authorize last version of tokenizer
      
      * Update version table
      
      * Fix conversion of spm tokenizers and fix some hub links
      
      * Bump tokenizers version to 0.10.1rc1
      
      * Add script to check tokenizers conversion with XNLI
      
      * Add some more mask_token lstrip support
      
      * Must modify mask_token in slow tokenizers too
      
      * Keep using the old method for Pegasus
      
      * add missing import
      Co-authored-by: default avatarAnthony MOI <m.anthony.moi@gmail.com>
      21b3922e