Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
2e4de762
"git@developer.sourcefind.cn:cnjsdfcy/simbricks.git" did not exist on "7dfdc0a5563abe80a85f9f7fa0c3b2ef458e7783"
Unverified
Commit
2e4de762
authored
Jun 03, 2020
by
Lysandre Debut
Committed by
GitHub
Jun 03, 2020
Browse files
Update encode documentation (#4751)
parent
ed4df855
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
2 deletions
+3
-2
src/transformers/tokenization_utils.py
src/transformers/tokenization_utils.py
+3
-2
No files found.
src/transformers/tokenization_utils.py
View file @
2e4de762
...
@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin):
...
@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin):
**
kwargs
**
kwargs
):
):
"""
"""
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Adds the model-specific
special tokens (such as beginning of sequence, end of sequence, sequence separator).
S
ame as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
If specifying ``add_special_tokens=False``, s
ame as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
Args:
Args:
text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):
text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment