• Arthur's avatar
    Nit-added-tokens (#26538) · 1a2e966c
    Arthur authored
    * fix stripping
    
    * nits
    
    * fix another test
    
    * styling
    
    * fix?
    
    * update
    
    * revert bad merge
    
    * found the bug
    
    * YES SIR
    
    * is that change really required?
    
    * make fast even faster
    
    * re order functions
    1a2e966c
tokenization_utils_fast.py 34.7 KB