Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
f349826a
Commit
f349826a
authored
Dec 17, 2019
by
Stefan Schweter
Browse files
model: fix cls and sep token for XLM-RoBERTa documentation
parent
d064009b
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
3 deletions
+3
-3
transformers/modeling_xlm_roberta.py
transformers/modeling_xlm_roberta.py
+3
-3
No files found.
transformers/modeling_xlm_roberta.py
View file @
f349826a
...
@@ -111,17 +111,17 @@ class XLMRobertaModel(RobertaModel):
...
@@ -111,17 +111,17 @@ class XLMRobertaModel(RobertaModel):
Last layer hidden-state of the first token of the sequence (classification token)
Last layer hidden-state of the first token of the sequence (classification token)
further processed by a Linear layer and a Tanh activation function. The Linear
further processed by a Linear layer and a Tanh activation function. The Linear
layer weights are trained from the next sentence prediction (classification)
layer weights are trained from the next sentence prediction (classification)
eo match pre-training, XLM-RoBERTa input sequence should be formatted with
[CLS] and [SEP]
tokens as follows:
eo match pre-training, XLM-RoBERTa input sequence should be formatted with
<s> and </s>
tokens as follows:
(a) For sequence pairs:
(a) For sequence pairs:
``tokens:
[CLS]
is this jack ##son ##ville ?
[SEP] [SEP]
no it is not .
[SEP]
``
``tokens:
<s>
is this jack ##son ##ville ?
</s> </s>
no it is not .
</s>
``
``token_type_ids: 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1``
``token_type_ids: 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1``
(b) For single sequences:
(b) For single sequences:
``tokens:
[CLS]
the dog is hairy .
[SEP]
``
``tokens:
<s>
the dog is hairy .
</s>
``
``token_type_ids: 0 0 0 0 0 0 0``
``token_type_ids: 0 0 0 0 0 0 0``
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment