Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
4a433e32
Unverified
Commit
4a433e32
authored
Dec 21, 2022
by
Arthur
Committed by
GitHub
Dec 21, 2022
Browse files
Add-warning-tokenizer (#20826)
* add fast not use warning * update
parent
76d02fea
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
8 additions
and
3 deletions
+8
-3
src/transformers/models/auto/tokenization_auto.py
src/transformers/models/auto/tokenization_auto.py
+8
-3
No files found.
src/transformers/models/auto/tokenization_auto.py
View file @
4a433e32
...
...
@@ -576,9 +576,14 @@ class AutoTokenizer:
tokenizer_class_name
,
tokenizer_fast_class_name
=
tokenizer_class_tuple
if
use_fast
and
tokenizer_fast_class_name
is
not
None
:
tokenizer_class
=
tokenizer_class_from_name
(
tokenizer_fast_class_name
)
if
use_fast
:
if
tokenizer_fast_class_name
is
not
None
:
tokenizer_class
=
tokenizer_class_from_name
(
tokenizer_fast_class_name
)
else
:
logger
.
warning
(
"`use_fast` is set to `True` but the tokenizer class does not have a fast version. "
" Falling back to the slow version."
)
if
tokenizer_class
is
None
:
tokenizer_class
=
tokenizer_class_from_name
(
tokenizer_class_name
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment