Skip to content

GitLab

Unverified Commit 245cdb46 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Fix barthez tokenizer (#9562)

parent 247a7b20
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
src/transformers/models/auto/tokenization_auto.py
View file @ 245cdb46
...@@ -199,7 +199,6 @@ TOKENIZER_MAPPING = OrderedDict( ...@@ -199,7 +199,6 @@ TOKENIZER_MAPPING = OrderedDict(
(AlbertConfig, (AlbertTokenizer, AlbertTokenizerFast)), (AlbertConfig, (AlbertTokenizer, AlbertTokenizerFast)),
(CamembertConfig, (CamembertTokenizer, CamembertTokenizerFast)), (CamembertConfig, (CamembertTokenizer, CamembertTokenizerFast)),
(PegasusConfig, (PegasusTokenizer, PegasusTokenizerFast)), (PegasusConfig, (PegasusTokenizer, PegasusTokenizerFast)),
(MBartConfig, (BarthezTokenizer, BarthezTokenizerFast)),
(MBartConfig, (MBartTokenizer, MBartTokenizerFast)), (MBartConfig, (MBartTokenizer, MBartTokenizerFast)),
(XLMRobertaConfig, (XLMRobertaTokenizer, XLMRobertaTokenizerFast)), (XLMRobertaConfig, (XLMRobertaTokenizer, XLMRobertaTokenizerFast)),
(MarianConfig, (MarianTokenizer, None)), (MarianConfig, (MarianTokenizer, None)),
...@@ -243,6 +242,7 @@ NO_CONFIG_TOKENIZER = [ ...@@ -243,6 +242,7 @@ NO_CONFIG_TOKENIZER = [
HerbertTokenizer, HerbertTokenizer,
HerbertTokenizerFast, HerbertTokenizerFast,
PhobertTokenizer, PhobertTokenizer,
BarthezTokenizer,
] ]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment