Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
49dc29bd
Commit
49dc29bd
authored
Feb 06, 2022
by
Jonathan Tow
Browse files
Bump Hendrycks `MATH` because of updated dataset
parent
26f0233f
Changes
15
Hide whitespace changes
Inline
Side-by-side
Showing
15 changed files
with
22 additions
and
8 deletions
+22
-8
lm_eval/tasks/hendrycks_math.py
lm_eval/tasks/hendrycks_math.py
+8
-8
tests/testdata/math_algebra-v1-greedy_until
tests/testdata/math_algebra-v1-greedy_until
+1
-0
tests/testdata/math_algebra-v1-res.json
tests/testdata/math_algebra-v1-res.json
+1
-0
tests/testdata/math_counting_and_prob-v1-greedy_until
tests/testdata/math_counting_and_prob-v1-greedy_until
+1
-0
tests/testdata/math_counting_and_prob-v1-res.json
tests/testdata/math_counting_and_prob-v1-res.json
+1
-0
tests/testdata/math_geometry-v1-greedy_until
tests/testdata/math_geometry-v1-greedy_until
+1
-0
tests/testdata/math_geometry-v1-res.json
tests/testdata/math_geometry-v1-res.json
+1
-0
tests/testdata/math_intermediate_algebra-v1-greedy_until
tests/testdata/math_intermediate_algebra-v1-greedy_until
+1
-0
tests/testdata/math_intermediate_algebra-v1-res.json
tests/testdata/math_intermediate_algebra-v1-res.json
+1
-0
tests/testdata/math_num_theory-v1-greedy_until
tests/testdata/math_num_theory-v1-greedy_until
+1
-0
tests/testdata/math_num_theory-v1-res.json
tests/testdata/math_num_theory-v1-res.json
+1
-0
tests/testdata/math_prealgebra-v1-greedy_until
tests/testdata/math_prealgebra-v1-greedy_until
+1
-0
tests/testdata/math_prealgebra-v1-res.json
tests/testdata/math_prealgebra-v1-res.json
+1
-0
tests/testdata/math_precalc-v1-greedy_until
tests/testdata/math_precalc-v1-greedy_until
+1
-0
tests/testdata/math_precalc-v1-res.json
tests/testdata/math_precalc-v1-res.json
+1
-0
No files found.
lm_eval/tasks/hendrycks_math.py
View file @
49dc29bd
...
...
@@ -18,7 +18,7 @@ class Math(Task):
def
download
(
self
):
if
not
(
self
.
DATASET_PATH
/
'test'
).
exists
()
or
not
(
self
.
DATASET_PATH
/
'done'
).
exists
():
sh
(
f
"mkdir -p
{
self
.
DATASET_PATH
}
"
)
download_file
(
"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar"
,
local_file
=
f
"
{
self
.
DATASET_PATH
}
.tar"
,
expected_checksum
=
"0
1256fd7cd5430596fdf07e6e6a5827111b5235b7ffed679c662a12f898932da
"
)
download_file
(
"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar"
,
local_file
=
f
"
{
self
.
DATASET_PATH
}
.tar"
,
expected_checksum
=
"0
fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac
"
)
sh
(
f
"""
tar -xf
{
self
.
DATASET_PATH
}
.tar -C data/ && touch
{
self
.
DATASET_PATH
/
'done'
}
rm
{
self
.
DATASET_PATH
}
.tar
...
...
@@ -285,42 +285,42 @@ class Math(Task):
class
MathAlgebra
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'algebra'
class
MathCountingAndProbability
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'counting_and_probability'
class
MathGeometry
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'geometry'
class
MathIntermediateAlgebra
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'intermediate_algebra'
class
MathNumberTheory
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'number_theory'
class
MathPrealgebra
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'prealgebra'
class
MathPrecalculus
(
Math
):
VERSION
=
0
VERSION
=
1
def
get_file_info
(
self
):
return
'precalculus'
tests/testdata/math_algebra-v1-greedy_until
0 → 100644
View file @
49dc29bd
f19182ce697a2c095d9e5b56ee6659dc38c93994b69ca75d7c3d3f5fd87572b4
\ No newline at end of file
tests/testdata/math_algebra-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_algebra"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_algebra"
:
1
}}
\ No newline at end of file
tests/testdata/math_counting_and_prob-v1-greedy_until
0 → 100644
View file @
49dc29bd
2aa9ae43ee9dbb2457525247d7b65358632c5eaa9cbfc40cf95a4f17f5d942ad
\ No newline at end of file
tests/testdata/math_counting_and_prob-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_counting_and_prob"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_counting_and_prob"
:
1
}}
\ No newline at end of file
tests/testdata/math_geometry-v1-greedy_until
0 → 100644
View file @
49dc29bd
46bc4cb219b6903397da782699a684bdbb982c0c954ff82e6beeed5c84878f42
\ No newline at end of file
tests/testdata/math_geometry-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_geometry"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_geometry"
:
1
}}
\ No newline at end of file
tests/testdata/math_intermediate_algebra-v1-greedy_until
0 → 100644
View file @
49dc29bd
d53c699de272d517ed7ad783b4e692302be9f9f97a8d4ac7a6541e538a7cabe0
\ No newline at end of file
tests/testdata/math_intermediate_algebra-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_intermediate_algebra"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_intermediate_algebra"
:
1
}}
\ No newline at end of file
tests/testdata/math_num_theory-v1-greedy_until
0 → 100644
View file @
49dc29bd
b920ccb507afdcf3ef6f4c04891913731e9f32ec914801791c6d9f8abf6e1897
\ No newline at end of file
tests/testdata/math_num_theory-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_num_theory"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_num_theory"
:
1
}}
\ No newline at end of file
tests/testdata/math_prealgebra-v1-greedy_until
0 → 100644
View file @
49dc29bd
752cdf343d7152e476b0273065024f6ea0e0f47ea385c6bdf9067736cb39724a
\ No newline at end of file
tests/testdata/math_prealgebra-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_prealgebra"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_prealgebra"
:
1
}}
\ No newline at end of file
tests/testdata/math_precalc-v1-greedy_until
0 → 100644
View file @
49dc29bd
bc834b06fd79473ca6fe38a51b714aad0bf0478c1b0eec787eca34dbdf69cb71
\ No newline at end of file
tests/testdata/math_precalc-v1-res.json
0 → 100644
View file @
49dc29bd
{
"results"
:
{
"math_precalc"
:
{
"acc"
:
0.0
,
"acc_stderr"
:
0.0
}},
"versions"
:
{
"math_precalc"
:
1
}}
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment