• Tanmay Laud's avatar
    Big Bird Fast Tokenizer implementation (#11075) · f7f87295
    Tanmay Laud authored
    
    
    * Added Big Bird Fast Tokenizer initial file
    
    * style fixes
    
    * flake fixes
    
    * Added big bird fast tokenizer to init files
    
    * Added big bird fast to Auto tokenization
    
    * fix styles
    
    * minor quality fixes
    
    * Added initial test code
    
    * Fix SpmConverter when precompiled_charsmap doesn't exist
    
    * fixed post processor
    
    * minor style fix
    
    * minor fix input names
    
    * Actually fix identity normalization
    
    * style
    
    * Added token type ids to fast tokenizer
    
    * style
    
    * flake fix
    
    * fix copies
    Co-authored-by: default avatarAnthony MOI <m.anthony.moi@gmail.com>
    f7f87295
dummy_tokenizers_objects.py 8.1 KB