• Patrick von Platen's avatar
    [Pegasus] Refactor Tokenizer (#8731) · 5ced23dc
    Patrick von Platen authored
    * refactor
    
    * further refactor
    
    * fix the rest tomorrow
    
    * save intermediate
    
    * finish slow tokenizer
    
    * make more tests pass
    
    * finish refactor
    
    * fix comment
    
    * clean further
    
    * fix name
    
    * fix naming
    
    * Update src/transformers/models/reformer/tokenization_reformer.py
    
    * Apply suggestions from code review
    
    * Apply suggestions from code review
    
    * refactor
    
    * fix init tokenizers
    
    * refactor
    
    * improve convert
    
    * refactor
    
    * correct convert slow tokenizer
    
    * final fix for Pegasus Tok
    
    * remove ipdb
    
    * improve links
    5ced23dc
test_tokenization_pegasus.py 3.89 KB