Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
0eb94c8b
Commit
0eb94c8b
authored
Aug 01, 2023
by
lintangsutawika
Browse files
process aggregate fix
parent
b2598de8
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
1 addition
and
4 deletions
+1
-4
lm_eval/evaluator.py
lm_eval/evaluator.py
+1
-4
No files found.
lm_eval/evaluator.py
View file @
0eb94c8b
...
...
@@ -386,9 +386,6 @@ def evaluate(
task_score
=
task
.
aggregation
()[
metric
](
items
)
results
[
task_name
][
metric
+
","
+
key
]
=
task_score
# if task_name not in benchmark_agg:
# benchmark[] = [task_score]
# Need to put back in results
# pythia | acc
# | perplexity
...
...
@@ -415,7 +412,7 @@ def evaluate(
if
stderr
is
not
None
:
results
[
task_name
][
metric
+
"_stderr"
+
","
+
key
]
=
stderr
(
items
)
if
not
bool
(
aggregate
):
if
bool
(
aggregate
):
for
group
in
aggregate
.
keys
():
for
metric
in
aggregate
[
group
].
keys
():
aggregate
[
group
][
metric
]
=
np
.
average
(
aggregate
[
group
][
metric
])
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment