Skip to content

GitLab

Unverified Commit a96edb85 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

GPT Neo configuration needs to be set to use GPT2 tokenizer (#10992)

parent bf0840ac
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
src/transformers/models/auto/tokenization_auto.py
View file @ a96edb85
...@@ -17,6 +17,7 @@ ...@@ -17,6 +17,7 @@
from collections import OrderedDict from collections import OrderedDict
from ... import GPTNeoConfig
from ...configuration_utils import PretrainedConfig from ...configuration_utils import PretrainedConfig
from ...file_utils import is_sentencepiece_available, is_tokenizers_available from ...file_utils import is_sentencepiece_available, is_tokenizers_available
from ...utils import logging from ...utils import logging
...@@ -264,6 +265,7 @@ TOKENIZER_MAPPING = OrderedDict( ...@@ -264,6 +265,7 @@ TOKENIZER_MAPPING = OrderedDict(
(BigBirdConfig, (BigBirdTokenizer, None)), (BigBirdConfig, (BigBirdTokenizer, None)),
(IBertConfig, (RobertaTokenizer, RobertaTokenizerFast)), (IBertConfig, (RobertaTokenizer, RobertaTokenizerFast)),
(Wav2Vec2Config, (Wav2Vec2CTCTokenizer, None)), (Wav2Vec2Config, (Wav2Vec2CTCTokenizer, None)),
(GPTNeoConfig, (GPT2Tokenizer, GPT2TokenizerFast)),
] ]
) )
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment