Skip to content

GitLab

Unverified Commit 15ef0dc5 authored by Nicolas Patry's avatar Nicolas Patry Committed by GitHub
Browse files

Enabling AutoTokenizer for HubertConfig. (#12198)

parent afa414d0
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
src/transformers/models/auto/tokenization_auto.py
View file @ 15ef0dc5
...@@ -81,6 +81,7 @@ from .configuration_auto import ( ...@@ -81,6 +81,7 @@ from .configuration_auto import (
FSMTConfig, FSMTConfig,
FunnelConfig, FunnelConfig,
GPT2Config, GPT2Config,
HubertConfig,
IBertConfig, IBertConfig,
LayoutLMConfig, LayoutLMConfig,
LEDConfig, LEDConfig,
...@@ -280,6 +281,7 @@ TOKENIZER_MAPPING = OrderedDict( ...@@ -280,6 +281,7 @@ TOKENIZER_MAPPING = OrderedDict(
(BigBirdConfig, (BigBirdTokenizer, BigBirdTokenizerFast)), (BigBirdConfig, (BigBirdTokenizer, BigBirdTokenizerFast)),
(IBertConfig, (RobertaTokenizer, RobertaTokenizerFast)), (IBertConfig, (RobertaTokenizer, RobertaTokenizerFast)),
(Wav2Vec2Config, (Wav2Vec2CTCTokenizer, None)), (Wav2Vec2Config, (Wav2Vec2CTCTokenizer, None)),
(HubertConfig, (Wav2Vec2CTCTokenizer, None)),
(GPTNeoConfig, (GPT2Tokenizer, GPT2TokenizerFast)), (GPTNeoConfig, (GPT2Tokenizer, GPT2TokenizerFast)),
(LukeConfig, (LukeTokenizer, None)), (LukeConfig, (LukeTokenizer, None)),
(BigBirdPegasusConfig, (PegasusTokenizer, PegasusTokenizerFast)), (BigBirdPegasusConfig, (PegasusTokenizer, PegasusTokenizerFast)),
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment