Commit 699bc7e8 authored by thomwolf's avatar thomwolf
Browse files

fix gpt-2 unk token test

parent 762ded9b
...@@ -57,4 +57,4 @@ It should build the static app that will be available under `/docs/_build/html` ...@@ -57,4 +57,4 @@ It should build the static app that will be available under `/docs/_build/html`
## Adding a new element to the tree (toc-tree) ## Adding a new element to the tree (toc-tree)
Accepted files are reStructuredText (.rst) and Markdown (.md). Create a file with its extension and put it Accepted files are reStructuredText (.rst) and Markdown (.md). Create a file with its extension and put it
in the source directory. You can then link it to the toc-tree by putting the filename without the extension. in the source directory. You can then link it to the toc-tree by putting the filename without the extension.
\ No newline at end of file
...@@ -177,7 +177,9 @@ class GPT2Tokenizer(PreTrainedTokenizer): ...@@ -177,7 +177,9 @@ class GPT2Tokenizer(PreTrainedTokenizer):
def _convert_token_to_id(self, token): def _convert_token_to_id(self, token):
""" Converts a token (str/unicode) in an id using the vocab. """ """ Converts a token (str/unicode) in an id using the vocab. """
return self.encoder.get(token) if token in self.encoder:
return self.encoder.get(token)
return self.encoder.get(self.unk_token)
def _convert_id_to_token(self, index): def _convert_id_to_token(self, index):
"""Converts an index (integer) in a token (string/unicode) using the vocab.""" """Converts an index (integer) in a token (string/unicode) using the vocab."""
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment