tests/test_tokenization_common.py · 49debe62fdc96e161f866dd8914d5915477bb742 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

Force pad_token_id to be set before padding for standard tokenizer (#3035) · c0135194
Patrick von Platen authored Mar 02, 2020
```
* force pad_token_id to be set before padding

* fix tests and forbid padding without having a padding_token_id set
```
c0135194

test_tokenization_common.py 29.9 KB