Skip to content

GitLab

Commit 4c12860f authored by LysandreJik's avatar LysandreJik
Browse files

Remove misleading documentation

parent 51ae2032
Hide whitespace changes
Inline Side-by-side
Showing with 0 additions and 1 deletion
transformers/tokenization_utils.py
View file @ 4c12860f
...@@ -628,7 +628,6 @@ class PreTrainedTokenizer(object): ...@@ -628,7 +628,6 @@ class PreTrainedTokenizer(object):
Take care of added tokens. Take care of added tokens.
text: The sequence to be encoded. text: The sequence to be encoded.
return_tokens_mapped_to_origin: (optional) Set to True to return the index of each token in the initial whitespace tokenization. (default False).
**kwargs: passed to the child `self.tokenize()` method **kwargs: passed to the child `self.tokenize()` method
""" """
def lowercase_text(t): def lowercase_text(t):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment