Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
85277915
Commit
85277915
authored
Jun 04, 2021
by
Leo Gao
Browse files
fix bug where interrupted pile download doesnt correctly resume
parent
17c825d1
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
4 deletions
+2
-4
lm_eval/tasks/pile.py
lm_eval/tasks/pile.py
+2
-4
No files found.
lm_eval/tasks/pile.py
View file @
85277915
...
...
@@ -19,10 +19,8 @@ class PilePerplexityTask(PerplexityTask, abc.ABC):
def
download
(
self
):
# TODO: separate pile val/test out by component so we don't have to scan the entire file once per set
os
.
makedirs
(
"data/pile/"
,
exist_ok
=
True
)
if
not
os
.
path
.
exists
(
self
.
VAL_PATH
):
download_file
(
"https://the-eye.eu/public/AI/pile/val.jsonl.zst"
,
self
.
VAL_PATH
)
if
not
os
.
path
.
exists
(
self
.
TEST_PATH
):
download_file
(
"https://the-eye.eu/public/AI/pile/test.jsonl.zst"
,
self
.
TEST_PATH
)
download_file
(
"https://the-eye.eu/public/AI/pile/val.jsonl.zst"
,
self
.
VAL_PATH
)
download_file
(
"https://the-eye.eu/public/AI/pile/test.jsonl.zst"
,
self
.
TEST_PATH
)
def
validation_docs
(
self
):
rdr
=
lm_dataformat
.
Reader
(
self
.
VAL_PATH
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment