tests/test_tokenization_t5.py · e78c1103385f2d2f9cd4980f61a8e71baa655356 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

T5Tokenizer adds EOS token if not already added (#5866) · 62449570
Sam Shleifer authored Aug 25, 2020
```
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
```
62449570

test_tokenization_t5.py 7.1 KB