Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
1bf3c505
Commit
1bf3c505
authored
Apr 15, 2023
by
kabbi159
Browse files
fix: add acc_norm in kobest_hellaswag
parent
cc7650d8
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
9 additions
and
1 deletion
+9
-1
lm_eval/tasks/kobest.py
lm_eval/tasks/kobest.py
+9
-1
No files found.
lm_eval/tasks/kobest.py
View file @
1bf3c505
...
...
@@ -248,20 +248,28 @@ class HellaSwag(MultipleChoiceTask):
def
process_results
(
self
,
doc
,
results
):
pred
=
np
.
argmax
(
results
)
gold
=
doc
[
"gold"
]
acc
=
1.
if
np
.
argmax
(
results
)
==
gold
else
0.
completion_len
=
np
.
array
([
float
(
len
(
i
))
for
i
in
doc
[
"choices"
]])
acc_norm
=
1.
if
np
.
argmax
(
results
/
completion_len
)
==
gold
else
0.
return
{
"acc"
:
pred
==
gold
,
"acc"
:
acc
,
"acc_norm"
:
acc_norm
,
"macro_f1"
:
(
gold
,
pred
)
}
def
higher_is_better
(
self
):
return
{
"acc"
:
True
,
"acc_norm"
:
True
,
"macro_f1"
:
True
}
def
aggregation
(
self
):
return
{
"acc"
:
mean
,
"acc_norm"
:
mean
,
"macro_f1"
:
macro_f1_score
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment