"vscode:/vscode.git/clone" did not exist on "9f1260971f041f4dcabf063ca2964847c3e5fc2a"
  • Sylvain Gugger's avatar
    Refactor `prepare_seq2seq_batch` (#9524) · 063d8d27
    Sylvain Gugger authored
    * Add target contextmanager and rework prepare_seq2seq_batch
    
    * Fix tests, treat BART and Barthez
    
    * Add last tokenizers
    
    * Fix test
    
    * Set src token before calling the superclass
    
    * Remove special behavior for T5
    
    * Remove needless imports
    
    * Remove needless asserts
    063d8d27
test_tokenization_common.py 141 KB