Commit 07f4cd73 authored by Rémi Louf's avatar Rémi Louf Committed by Julien Chaumond
Browse files

update function to add special tokens

Since I started my PR the `add_special_token_single_sequence` function
has been deprecated for another; I replaced it with the new function.
parent 5c877fe9
...@@ -139,11 +139,11 @@ def encode_for_summarization(story_lines, summary_lines, tokenizer): ...@@ -139,11 +139,11 @@ def encode_for_summarization(story_lines, summary_lines, tokenizer):
sentences. sentences.
""" """
story_lines_token_ids = [ story_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line)) tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in story_lines for line in story_lines
] ]
summary_lines_token_ids = [ summary_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line)) tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in summary_lines for line in summary_lines
] ]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment