Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
5d912e7e
Commit
5d912e7e
authored
Apr 04, 2020
by
Julien Chaumond
Browse files
Tweak typing for #3566
parent
94eb68d7
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
2 deletions
+3
-2
src/transformers/tokenization_bert_japanese.py
src/transformers/tokenization_bert_japanese.py
+3
-2
No files found.
src/transformers/tokenization_bert_japanese.py
View file @
5d912e7e
...
@@ -19,6 +19,7 @@ import collections
...
@@ -19,6 +19,7 @@ import collections
import
logging
import
logging
import
os
import
os
import
unicodedata
import
unicodedata
from
typing
import
Optional
from
.tokenization_bert
import
BasicTokenizer
,
BertTokenizer
,
WordpieceTokenizer
,
load_vocab
from
.tokenization_bert
import
BasicTokenizer
,
BertTokenizer
,
WordpieceTokenizer
,
load_vocab
...
@@ -165,10 +166,10 @@ class BertJapaneseTokenizer(BertTokenizer):
...
@@ -165,10 +166,10 @@ class BertJapaneseTokenizer(BertTokenizer):
return
split_tokens
return
split_tokens
class
MecabTokenizer
(
object
)
:
class
MecabTokenizer
:
"""Runs basic tokenization with MeCab morphological parser."""
"""Runs basic tokenization with MeCab morphological parser."""
def
__init__
(
self
,
do_lower_case
=
False
,
never_split
=
None
,
normalize_text
=
True
,
mecab_option
=
None
):
def
__init__
(
self
,
do_lower_case
=
False
,
never_split
=
None
,
normalize_text
=
True
,
mecab_option
:
Optional
[
str
]
=
None
):
"""Constructs a MecabTokenizer.
"""Constructs a MecabTokenizer.
Args:
Args:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment