Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
b9e55312
Commit
b9e55312
authored
Dec 13, 2023
by
lintangsutawika
Browse files
add alt prompts for truthfulqa
parent
edb21d21
Changes
6
Hide whitespace changes
Inline
Side-by-side
Showing
6 changed files
with
84 additions
and
0 deletions
+84
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_01/truthfulqa_mc1.yaml
...tive_worlds/prompt_variation/style_01/truthfulqa_mc1.yaml
+17
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_01/truthfulqa_mc2.yaml
...tive_worlds/prompt_variation/style_01/truthfulqa_mc2.yaml
+11
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_02/truthfulqa_mc1.yaml
...tive_worlds/prompt_variation/style_02/truthfulqa_mc1.yaml
+17
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_02/truthfulqa_mc2.yaml
...tive_worlds/prompt_variation/style_02/truthfulqa_mc2.yaml
+11
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_03/truthfulqa_mc1.yaml
...tive_worlds/prompt_variation/style_03/truthfulqa_mc1.yaml
+17
-0
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_03/truthfulqa_mc2.yaml
...tive_worlds/prompt_variation/style_03/truthfulqa_mc2.yaml
+11
-0
No files found.
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_01/truthfulqa_mc1.yaml
0 → 100644
View file @
b9e55312
task
:
truthfulqa_mc1_alt_pv_01
dataset_path
:
truthful_qa
dataset_name
:
multiple_choice
output_type
:
multiple_choice
training_split
:
null
validation_split
:
validation
test_split
:
null
num_fewshot
:
0
doc_to_text
:
"
{{question}}"
doc_to_target
:
0
doc_to_choice
:
"
{{mc1_targets.choices}}"
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_01/truthfulqa_mc2.yaml
0 → 100644
View file @
b9e55312
include
:
truthfulqa_mc1.yaml
task
:
truthfulqa_mc2_alt_pv_01
doc_to_target
:
0
doc_to_choice
:
"
{{mc2_targets.choices}}"
process_results
:
!function
../../../utils.process_results_mc2
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_02/truthfulqa_mc1.yaml
0 → 100644
View file @
b9e55312
task
:
truthfulqa_mc1_alt_pv_02
dataset_path
:
truthful_qa
dataset_name
:
multiple_choice
output_type
:
multiple_choice
training_split
:
null
validation_split
:
validation
test_split
:
null
num_fewshot
:
0
doc_to_choice
:
"
Q:
{{mc2_targets.choices}}
\n
A:"
doc_to_target
:
0
doc_to_choice
:
"
{{mc1_targets.choices}}"
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_02/truthfulqa_mc2.yaml
0 → 100644
View file @
b9e55312
include
:
truthfulqa_mc1.yaml
task
:
truthfulqa_mc2_alt_pv_02
doc_to_target
:
0
doc_to_choice
:
"
{{mc2_targets.choices}}"
process_results
:
!function
../../../utils.process_results_mc2
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_03/truthfulqa_mc1.yaml
0 → 100644
View file @
b9e55312
task
:
truthfulqa_mc1_alt_pv_03
dataset_path
:
truthful_qa
dataset_name
:
multiple_choice
output_type
:
multiple_choice
training_split
:
null
validation_split
:
validation
test_split
:
null
num_fewshot
:
0
doc_to_choice
:
"
Question:
{{mc2_targets.choices}}
\n
Answer:"
doc_to_target
:
0
doc_to_choice
:
"
{{mc1_targets.choices}}"
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/truthfulqa/alternative_worlds/prompt_variation/style_03/truthfulqa_mc2.yaml
0 → 100644
View file @
b9e55312
include
:
truthfulqa_mc1.yaml
task
:
truthfulqa_mc2_alt_pv_03
doc_to_target
:
0
doc_to_choice
:
"
{{mc2_targets.choices}}"
process_results
:
!function
../../../utils.process_results_mc2
should_decontaminate
:
True
doc_to_decontamination_query
:
question
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment