Unverified Commit 3c39c07f authored by SaulLu's avatar SaulLu Committed by GitHub
Browse files

fix `word_to_tokens` docstring format (#20450)

* fix docstring

* fix 2

* add details
parent a547d5bd
...@@ -475,8 +475,10 @@ class BatchEncoding(UserDict): ...@@ -475,8 +475,10 @@ class BatchEncoding(UserDict):
or 1) the provided word index belongs to. or 1) the provided word index belongs to.
Returns: Returns:
Optional [`~tokenization_utils_base.TokenSpan`] Span of tokens in the encoded sequence. Returns `None` if ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns
no tokens correspond to the word. `None` if no tokens correspond to the word. This can happen especially when the token is a special token
that has been used to format the tokenization. For example when we add a class token at the very beginning
of the tokenization.
""" """
if not self._encodings: if not self._encodings:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment