Commit 83446a88 authored by Bram Vanroy's avatar Bram Vanroy Committed by Lysandre Debut
Browse files

Use _pad_token of pad_token_id

Requesting pad_token_id would cause an error message when it is None. Use private _pad_token instead.
parent 9fde13a3
......@@ -999,7 +999,7 @@ class PreTrainedTokenizer(object):
padded_value = value
# verify that the tokenizer has a pad_token_id
if key != "input_len" and self.pad_token_id is not None:
if key != "input_len" and self._pad_token is not None:
# Padding handle
padded_value = [
v + [self.pad_token_id if key == "input_ids" else 1] * (max_seq_len - len(v))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment