src/transformers/tokenization_utils.py · c0135194ebc5de4b1bbef98b31f9c457a0bf746a · chenpangpang / transformers · GitLab

Find file Blame History Permalink

Force pad_token_id to be set before padding for standard tokenizer (#3035) · c0135194
Patrick von Platen authored Mar 02, 2020
```
* force pad_token_id to be set before padding

* fix tests and forbid padding without having a padding_token_id set
```
c0135194

tokenization_utils.py 90.8 KB