Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
c2f12474
Commit
c2f12474
authored
Feb 09, 2022
by
Stephen Hogg
Browse files
exponentiate unanswerable score; rename dict key
parent
7ed5e29e
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
3 deletions
+4
-3
lm_eval/tasks/qasper.py
lm_eval/tasks/qasper.py
+4
-3
No files found.
lm_eval/tasks/qasper.py
View file @
c2f12474
...
...
@@ -22,6 +22,7 @@ https://arxiv.org/abs/2105.03011
bibsource = {dblp computer science bibliography, https://dblp.org}
}
"""
from
math
import
exp
from
lm_eval.base
import
rf
from
lm_eval.metrics
import
f1_score
from
.common
import
HFTask
...
...
@@ -101,7 +102,7 @@ class QASPER(HFTask):
# Handle unanswerability first
unanswerable_gold
=
doc
[
"answer_type"
]
==
"unanswerable"
unanswerable_pred
=
unanswerable
>
1
-
unanswerable
unanswerable_pred
=
exp
(
unanswerable
)
>
1
-
exp
(
unanswerable
)
res_dict
[
"f1_un"
]
=
(
unanswerable_gold
,
unanswerable_pred
)
# Handle yes/no questions
...
...
@@ -113,14 +114,14 @@ class QASPER(HFTask):
# Handle completions
if
doc
[
"answer_type"
]
==
"free form answer"
:
pass
res_dict
[
"f1_ab"
]
=
None
return
res_dict
def
aggregation
(
self
):
return
{
"f1_un"
:
f1_score
,
"f1_yn"
:
f1_score
,
"f1_
fr
"
:
f1_score
,
"f1_
ab
"
:
f1_score
,
"f1_ex"
:
f1_score
,
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment