"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "b9ee87f5c730d72b326ef65089a574a0b519e827"
Unverified Commit 0e84559d authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Correct `special_tokens_mask` when `add_special_tokens=False` (#2965)

Don't know of a use case where that would be useful, but this is more consistent
parent 92487a1d
...@@ -1233,7 +1233,10 @@ class PreTrainedTokenizer(object): ...@@ -1233,7 +1233,10 @@ class PreTrainedTokenizer(object):
token_type_ids = [0] * len(ids) + ([1] * len(pair_ids) if pair else []) token_type_ids = [0] * len(ids) + ([1] * len(pair_ids) if pair else [])
if return_special_tokens_mask: if return_special_tokens_mask:
encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids) if add_special_tokens:
encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
else:
encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
encoded_inputs["input_ids"] = sequence encoded_inputs["input_ids"] = sequence
if return_token_type_ids: if return_token_type_ids:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment