• Daniel Stancl's avatar
    Add `BlenderbotTokenizerFast` (#13720) · d37f1fb8
    Daniel Stancl authored
    * Add the support for the fast (rust) implementation of BlenbderbotTokenizer
    
    * Fix a converter and a typo in a doc
    
    * Apply the patil-suraj's suggestion
    
    * (Nitpick) Fast tokenization -> Fast Tokenization in doc
    
    * Apply the SaulLu's suggestion
    
    * Apply Narsil's suggestion to fix test pipelines
    
    * Add encoder_no_repeat_ngram_size according to the Narsil's suggestion
    
    * Revert the last (unnecessary) commit
    
    * Override pipeline config for Blenderbot to allow for larger pos. emb.
    
    * make fix-copies
    d37f1fb8
tokenization_blenderbot.py 4.06 KB