tests/test_tokenization_gpt2.py · 063d8d27f4e1d089dc76f22e378b86b219167e3b · chenpangpang / transformers

Refactor `prepare_seq2seq_batch` (#9524) · 063d8d27

Sylvain Gugger authored Jan 12, 2021

* Add target contextmanager and rework prepare_seq2seq_batch

* Fix tests, treat BART and Barthez

* Add last tokenizers

* Fix test

* Set src token before calling the superclass

* Remove special behavior for T5

* Remove needless imports

* Remove needless asserts

063d8d27

test_tokenization_gpt2.py 6.7 KB

Replace test_tokenization_gpt2.py