Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
cb7b77a8
"...csrc/git@developer.sourcefind.cn:zhaoyu6/sglang.git" did not exist on "a55cb8b2f86cf5c28a4972f3df765dbd77e19947"
Commit
cb7b77a8
authored
Oct 30, 2019
by
VictorSanh
Committed by
Lysandre Debut
Nov 27, 2019
Browse files
fix some typos
parent
84a0b522
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
3 deletions
+3
-3
transformers/tokenization_xlm.py
transformers/tokenization_xlm.py
+3
-3
No files found.
transformers/tokenization_xlm.py
View file @
cb7b77a8
...
...
@@ -12,7 +12,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tokenization classes for
OpenAI GPT
."""
"""Tokenization classes for
XLM
."""
from
__future__
import
(
absolute_import
,
division
,
print_function
,
unicode_literals
)
...
...
@@ -758,9 +758,9 @@ class XLMTokenizer(PreTrainedTokenizer):
"""
Build model inputs from a sequence or a pair of sequence for sequence classification tasks
by concatenating and adding special tokens.
A
RoBERTa
sequence has the following format:
A
XLM
sequence has the following format:
single sequence: <s> X </s>
pair of sequences: <s> A </s>
</s>
B </s>
pair of sequences: <s> A </s> B </s>
"""
if
token_ids_1
is
None
:
return
[
self
.
cls_token_id
]
+
token_ids_0
+
[
self
.
sep_token_id
]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment