• Nicolas Patry's avatar
    Enabling `tokenizers` upgrade. (#14941) · 08cb5718
    Nicolas Patry authored
    * Enabling `tokenizers` upgrade.
    
    * Moved ugly comment.
    
    * Tokenizers==0.11.1 needs an update to keep borrow checker
    
    happy in highly contiguous calls.
    
    * Support both 0.11.1 and 0.11.0
    08cb5718
setup.py 13.1 KB