"git@developer.sourcefind.cn:cnjsdfcy/simbricks.git" did not exist on "7dfdc0a5563abe80a85f9f7fa0c3b2ef458e7783"
Unverified Commit 2e4de762 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Update encode documentation (#4751)

parent ed4df855
...@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin): ...@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin):
**kwargs **kwargs
): ):
""" """
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Adds the model-specific
special tokens (such as beginning of sequence, end of sequence, sequence separator).
Same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``. If specifying ``add_special_tokens=False``, same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
Args: Args:
text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`): text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment