Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
2a358f45
Unverified
Commit
2a358f45
authored
Oct 01, 2020
by
Sam Shleifer
Committed by
GitHub
Oct 01, 2020
Browse files
[s2s] fix nltk pytest race condition with FileLock (#7515)
parent
72d363d9
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
4 deletions
+5
-4
examples/seq2seq/sentence_splitter.py
examples/seq2seq/sentence_splitter.py
+5
-4
No files found.
examples/seq2seq/sentence_splitter.py
View file @
2a358f45
import
re
import
re
from
filelock
import
FileLock
try
:
try
:
import
nltk
import
nltk
...
@@ -9,13 +11,12 @@ except (ImportError, ModuleNotFoundError):
...
@@ -9,13 +11,12 @@ except (ImportError, ModuleNotFoundError):
NLTK_AVAILABLE
=
False
NLTK_AVAILABLE
=
False
if
NLTK_AVAILABLE
:
if
NLTK_AVAILABLE
:
try
:
with
FileLock
(
"a_random_string"
)
as
lock
:
nltk
.
download
(
"punkt"
,
quiet
=
True
)
nltk
.
download
(
"punkt"
,
quiet
=
True
)
except
FileExistsError
:
# multiprocessing race condition
pass
def
add_newline_to_end_of_each_sentence
(
x
:
str
)
->
str
:
def
add_newline_to_end_of_each_sentence
(
x
:
str
)
->
str
:
"""This was added to get rougeLsum scores matching published rougeL scores for BART and PEGASUS."""
re
.
sub
(
"<n>"
,
""
,
x
)
# remove pegasus newline char
re
.
sub
(
"<n>"
,
""
,
x
)
# remove pegasus newline char
assert
NLTK_AVAILABLE
,
"nltk must be installed to separate newlines betwee sentences. (pip install nltk)"
assert
NLTK_AVAILABLE
,
"nltk must be installed to separate newlines betwee
n
sentences. (pip install nltk)"
return
"
\n
"
.
join
(
nltk
.
sent_tokenize
(
x
))
return
"
\n
"
.
join
(
nltk
.
sent_tokenize
(
x
))
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment