"...guests/git@developer.sourcefind.cn:cnjsdfcy/simbricks.git" did not exist on "530a84e4506854cdc19d2ff8341478acac3b8974"
Unverified Commit 2e4de762 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Update encode documentation (#4751)

parent ed4df855
...@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin): ...@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin):
**kwargs **kwargs
): ):
""" """
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Adds the model-specific
special tokens (such as beginning of sequence, end of sequence, sequence separator).
Same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``. If specifying ``add_special_tokens=False``, same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
Args: Args:
text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`): text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment