Skip to content

GitLab

Unverified Commit c4734840 authored by Thomas Wolf's avatar Thomas Wolf Committed by GitHub
Browse files

Fix the tokenization warning noted in #5505 (#5550) 

* fix warning

* style and quality
parent 1bbc28be
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
src/transformers/tokenization_utils.py
View file @ c4734840
...@@ -588,8 +588,8 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase): ...@@ -588,8 +588,8 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase):
first_ids, first_ids,
second_ids, second_ids,
add_special_tokens=add_special_tokens, add_special_tokens=add_special_tokens,
padding_strategy=PaddingStrategy.DO_NOT_PAD, # we pad in batch afterward padding=PaddingStrategy.DO_NOT_PAD.value, # we pad in batch afterward
truncation_strategy=truncation_strategy, truncation=truncation_strategy.value,
max_length=max_length, max_length=max_length,
stride=stride, stride=stride,
pad_to_multiple_of=None, # we pad in batch afterward pad_to_multiple_of=None, # we pad in batch afterward
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment