Commit 07f4cd73 authored by Rémi Louf's avatar Rémi Louf Committed by Julien Chaumond
Browse files

update function to add special tokens

Since I started my PR the `add_special_token_single_sequence` function
has been deprecated for another; I replaced it with the new function.
parent 5c877fe9
......@@ -139,11 +139,11 @@ def encode_for_summarization(story_lines, summary_lines, tokenizer):
sentences.
"""
story_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line))
tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in story_lines
]
summary_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line))
tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in summary_lines
]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment