Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
d6c08b07
Commit
d6c08b07
authored
Sep 10, 2020
by
Julien Chaumond
Browse files
[AutoTokenizer] Correct error message
parent
db38f7ce
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
1 deletion
+3
-1
src/transformers/tokenization_auto.py
src/transformers/tokenization_auto.py
+3
-1
No files found.
src/transformers/tokenization_auto.py
View file @
d6c08b07
...
@@ -234,7 +234,9 @@ class AutoTokenizer:
...
@@ -234,7 +234,9 @@ class AutoTokenizer:
tokenizer_class_candidate
=
config
.
tokenizer_class
tokenizer_class_candidate
=
config
.
tokenizer_class
tokenizer_class
=
globals
().
get
(
tokenizer_class_candidate
)
tokenizer_class
=
globals
().
get
(
tokenizer_class_candidate
)
if
tokenizer_class
is
None
:
if
tokenizer_class
is
None
:
raise
ValueError
(
"Tokenizer class {} does not exist or is not currently imported."
)
raise
ValueError
(
"Tokenizer class {} does not exist or is not currently imported."
.
format
(
tokenizer_class_candidate
)
)
return
tokenizer_class
.
from_pretrained
(
pretrained_model_name_or_path
,
*
inputs
,
**
kwargs
)
return
tokenizer_class
.
from_pretrained
(
pretrained_model_name_or_path
,
*
inputs
,
**
kwargs
)
# if model is an encoder decoder, the encoder tokenizer class is used by default
# if model is an encoder decoder, the encoder tokenizer class is used by default
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment