Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
2e3452af
"docs/vscode:/vscode.git/clone" did not exist on "0e91f885c3c2702dfe6c3f4f81518e8052d59804"
Unverified
Commit
2e3452af
authored
Nov 02, 2022
by
Ripose
Committed by
GitHub
Nov 02, 2022
Browse files
make sentencepiece import conditional in bertjapanesetokenizer (#20012)
parent
8827e1b2
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
3 deletions
+6
-3
src/transformers/models/bert_japanese/tokenization_bert_japanese.py
...ormers/models/bert_japanese/tokenization_bert_japanese.py
+6
-3
No files found.
src/transformers/models/bert_japanese/tokenization_bert_japanese.py
View file @
2e3452af
...
@@ -21,11 +21,14 @@ import os
...
@@ -21,11 +21,14 @@ import os
import
unicodedata
import
unicodedata
from
typing
import
Any
,
Dict
,
List
,
Optional
,
Tuple
from
typing
import
Any
,
Dict
,
List
,
Optional
,
Tuple
import
sentencepiece
as
spm
from
...tokenization_utils
import
PreTrainedTokenizer
,
_is_control
,
_is_punctuation
,
_is_whitespace
from
...tokenization_utils
import
PreTrainedTokenizer
,
_is_control
,
_is_punctuation
,
_is_whitespace
from
...utils
import
logging
from
...utils
import
is_sentencepiece_available
,
logging
if
is_sentencepiece_available
():
import
sentencepiece
as
spm
else
:
spm
=
None
logger
=
logging
.
get_logger
(
__name__
)
logger
=
logging
.
get_logger
(
__name__
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment