src/transformers/tokenization_utils.py · cc6775cdf5b20ad382613d3bdbf0dd8364d23219 · chenpangpang / transformers

Fix max_length not taken into account when using pad_to_max_length on fast tokenizers (#2961) · cc6775cd

Funtowicz Morgan authored Feb 22, 2020



* enable_padding should pad up to max_length if set.
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Added more testing on padding.
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

cc6775cd

tokenization_utils.py 84.8 KB

Replace tokenization_utils.py