Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
a15562e1
Unverified
Commit
a15562e1
authored
Sep 03, 2019
by
Shijie Wu
Committed by
GitHub
Sep 03, 2019
Browse files
Fix reference of import when called for the second time
parent
0287d264
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
0 deletions
+4
-0
pytorch_transformers/tokenization_xlm.py
pytorch_transformers/tokenization_xlm.py
+4
-0
No files found.
pytorch_transformers/tokenization_xlm.py
View file @
a15562e1
...
...
@@ -705,6 +705,8 @@ class XLMTokenizer(PreTrainedTokenizer):
try
:
if
'pythainlp'
not
in
sys
.
modules
:
from
pythainlp.tokenize
import
word_tokenize
as
th_word_tokenize
else
:
th_word_tokenize
=
sys
.
modules
[
'pythainlp'
].
word_tokenize
except
(
AttributeError
,
ImportError
)
as
e
:
logger
.
error
(
"Make sure you install PyThaiNLP (https://github.com/PyThaiNLP/pythainlp) with the following steps"
)
logger
.
error
(
"1. pip install pythainlp"
)
...
...
@@ -714,6 +716,8 @@ class XLMTokenizer(PreTrainedTokenizer):
try
:
if
'jieba'
not
in
sys
.
modules
:
import
jieba
else
:
jieba
=
sys
.
modules
[
'jieba'
]
except
(
AttributeError
,
ImportError
)
as
e
:
logger
.
error
(
"Make sure you install Jieba (https://github.com/fxsjy/jieba) with the following steps"
)
logger
.
error
(
"1. pip install jieba"
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment