Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
5699c7ab
Commit
5699c7ab
authored
Jul 13, 2023
by
lintangsutawika
Browse files
update winogrande format
parent
b7449cd9
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
8 deletions
+8
-8
lm_eval/tasks/winogrande/preprocess_winogrande.py
lm_eval/tasks/winogrande/preprocess_winogrande.py
+6
-6
lm_eval/tasks/winogrande/winogrande.yaml
lm_eval/tasks/winogrande/winogrande.yaml
+2
-2
No files found.
lm_eval/tasks/winogrande/preprocess_winogrande.py
View file @
5699c7ab
def
doc_to_text
(
doc
):
def
doc_to_text
(
doc
):
idx
=
doc
[
"sentence"
].
index
(
"_"
)
answer_to_num
=
{
"1"
:
0
,
"2"
:
1
}
options
=
[
doc
[
"option1"
],
doc
[
"option2"
]]
return
answer_to_num
[
doc
[
"answer"
]]
return
[
doc
[
"sentence"
][:
idx
]
+
opt
for
opt
in
options
]
def
doc_to_target
(
doc
):
def
doc_to_target
(
doc
):
...
@@ -9,6 +8,7 @@ def doc_to_target(doc):
...
@@ -9,6 +8,7 @@ def doc_to_target(doc):
return
doc
[
"sentence"
][
idx
:].
strip
()
return
doc
[
"sentence"
][
idx
:].
strip
()
def
gold_alias
(
doc
):
def
doc_to_choice
(
doc
):
answer_to_num
=
{
"1"
:
0
,
"2"
:
1
}
idx
=
doc
[
"sentence"
].
index
(
"_"
)
return
answer_to_num
[
doc
[
"answer"
]]
options
=
[
doc
[
"option1"
],
doc
[
"option2"
]]
return
[
doc
[
"sentence"
][:
idx
]
+
opt
for
opt
in
options
]
lm_eval/tasks/winogrande/winogrande.yaml
View file @
5699c7ab
...
@@ -4,9 +4,9 @@ dataset_name: winogrande_xl
...
@@ -4,9 +4,9 @@ dataset_name: winogrande_xl
output_type
:
multiple_choice
output_type
:
multiple_choice
training_split
:
train
training_split
:
train
validation_split
:
validation
validation_split
:
validation
doc_to_target
:
!function
preprocess_winogrande.doc_to_target
doc_to_text
:
!function
preprocess_winogrande.doc_to_text
doc_to_text
:
!function
preprocess_winogrande.doc_to_text
gold_alias
:
!function
preprocess_winogrande.gold_alias
doc_to_target
:
!function
preprocess_winogrande.doc_to_target
doc_to_choice
:
!function
preprocess_winogrande.doc_to_choice
metric_list
:
metric_list
:
-
metric
:
acc
-
metric
:
acc
aggregation
:
mean
aggregation
:
mean
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment