Skip to content

GitLab

"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "49bee0aea44ef29c08d48f818f356275ef223da8"
Unverified Commit 7419d807 authored by Thomas Wang's avatar Thomas Wang Committed by GitHub
Browse files

Declare __len__ method in PreTrainedTokenizerBase (#21210)

parent ef530175
Show whitespace changes
Inline Side-by-side
Showing with 3 additions and 0 deletions
src/transformers/tokenization_utils_base.py
View file @ 7419d807
...@@ -1578,6 +1578,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin): ...@@ -1578,6 +1578,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
f" special_tokens={self.special_tokens_map_extended})" f" special_tokens={self.special_tokens_map_extended})"
) )
def __len__(self) -> int:
raise NotImplementedError()
def get_vocab(self) -> Dict[str, int]: def get_vocab(self) -> Dict[str, int]:
""" """
Returns the vocabulary as a dictionary of token to index. Returns the vocabulary as a dictionary of token to index.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment