Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
f034974a
Commit
f034974a
authored
Jun 06, 2023
by
Oleh Shliazhko
Browse files
address review comments
parent
3a424af4
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
2 deletions
+6
-2
lm_eval/tasks/hendrycks_test.py
lm_eval/tasks/hendrycks_test.py
+6
-2
No files found.
lm_eval/tasks/hendrycks_test.py
View file @
f034974a
...
...
@@ -125,9 +125,13 @@ class GeneralHendrycksTest(MultipleChoiceTask):
def
test_docs
(
self
):
return
map
(
self
.
_process_doc
,
self
.
dataset
[
"test"
])
def
_format_subject
(
self
,
subject
):
words
=
subject
.
split
(
"_"
)
return
" "
.
join
(
words
)
def
fewshot_context
(
self
,
doc
,
num_fewshot
,
**
kwargs
):
subject
=
self
.
DATASET_NAME
description
=
f
"The following are multiple choice questions (with answers) about
{
s
ubject
}
."
description
=
f
"The following are multiple choice questions (with answers) about
{
s
elf
.
_format_subject
(
subject
)
}
."
kwargs
[
"description"
]
=
description
return
super
().
fewshot_context
(
doc
=
doc
,
num_fewshot
=
num_fewshot
,
**
kwargs
)
...
...
@@ -142,7 +146,7 @@ class GeneralHendrycksTest(MultipleChoiceTask):
Answer:
"""
question
=
doc
[
"question"
]
question
=
doc
[
"question"
]
.
strip
()
choices
=
""
.
join
(
[
f
"
{
key
}
.
{
choice
}
\n
"
for
key
,
choice
in
zip
(
keys
,
doc
[
"choices"
])]
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment