Skip to content

GitLab

Commit a2d4950f authored by Maxpa1n's avatar Maxpa1n Committed by Julien Chaumond
Browse files

fix annotation

parent 9f995b99
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 4 deletions
pytorch_transformers/tokenization_xlnet.py
View file @ a2d4950f
...@@ -183,8 +183,8 @@ class XLNetTokenizer(PreTrainedTokenizer): ...@@ -183,8 +183,8 @@ class XLNetTokenizer(PreTrainedTokenizer):
def add_special_tokens_single_sentence(self, token_ids): def add_special_tokens_single_sentence(self, token_ids):
""" """
Adds special tokens to a sequence pair for sequence classification tasks. Adds special tokens to a sequence for sequence classification tasks.
An XLNet sequence pair has the following format: A [SEP] B [SEP][CLS] An XLNet sequence has the following format: X [SEP][CLS]
""" """
sep = [self.sep_token_id] sep = [self.sep_token_id]
cls = [self.cls_token_id] cls = [self.cls_token_id]
...@@ -192,8 +192,8 @@ class XLNetTokenizer(PreTrainedTokenizer): ...@@ -192,8 +192,8 @@ class XLNetTokenizer(PreTrainedTokenizer):
def add_special_tokens_sentences_pair(self, token_ids_0, token_ids_1): def add_special_tokens_sentences_pair(self, token_ids_0, token_ids_1):
""" """
Adds special tokens to a sequence for sequence classification tasks. Adds special tokens to a sequence pair for sequence classification tasks.
An XLNet sequence has the following format: X [SEP][CLS] An XLNet sequence pair has the following format: A [SEP] B [SEP][CLS]
""" """
sep = [self.sep_token_id] sep = [self.sep_token_id]
cls = [self.cls_token_id] cls = [self.cls_token_id]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment