Skip to content

GitLab

Commit 894812c6 authored by Julien Chaumond's avatar Julien Chaumond
Browse files

Fixup mapping

parent b20f11d4
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
src/transformers/tokenization_auto.py
View file @ 894812c6
...@@ -60,8 +60,8 @@ TOKENIZER_MAPPING = OrderedDict( ...@@ -60,8 +60,8 @@ TOKENIZER_MAPPING = OrderedDict(
(DistilBertConfig, DistilBertTokenizer), (DistilBertConfig, DistilBertTokenizer),
(AlbertConfig, AlbertTokenizer), (AlbertConfig, AlbertTokenizer),
(CamembertConfig, CamembertTokenizer), (CamembertConfig, CamembertTokenizer),
(RobertaConfig, XLMRobertaTokenizer), (RobertaConfig, RobertaTokenizer),
(XLMRobertaConfig, RobertaTokenizer), (XLMRobertaConfig, XLMRobertaTokenizer),
(BertConfig, BertTokenizer), (BertConfig, BertTokenizer),
(OpenAIGPTConfig, OpenAIGPTTokenizer), (OpenAIGPTConfig, OpenAIGPTTokenizer),
(GPT2Config, GPT2Tokenizer), (GPT2Config, GPT2Tokenizer),
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment