Skip to content

GitLab

Commit 3927d775 authored by LysandreJik's avatar LysandreJik
Browse files

Updated the GLUE pre-processing method

parent 0ea82b24
Show whitespace changes
Inline Side-by-side
Showing with 3 additions and 2 deletions
examples/utils_glue.py
View file @ 3927d775
...@@ -409,8 +409,9 @@ def convert_examples_to_features(examples, label_list, max_seq_length, ...@@ -409,8 +409,9 @@ def convert_examples_to_features(examples, label_list, max_seq_length,
example.text_a, example.text_a,
example.text_b, example.text_b,
add_special_tokens=True, add_special_tokens=True,
output_mask=True, output_token_type=True,
max_length=max_seq_length max_length=max_seq_length,
truncate_first_sequence=True # We're truncating the first sequence as a priority
) )
input_ids, segment_ids = inputs["input_ids"], inputs["output_token_type"] input_ids, segment_ids = inputs["input_ids"], inputs["output_token_type"]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment