Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
bbb8386c
You need to sign in or sign up before continuing.
Commit
bbb8386c
authored
Apr 16, 2024
by
lintangsutawika
Browse files
removed alt worlds prompts
parent
3e5e9da2
Changes
1000
Hide whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
0 additions
and
125 deletions
+0
-125
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_public_relations.yaml
...ds/output_variation/style_03/a/mmlu_public_relations.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_security_studies.yaml
...ds/output_variation/style_03/a/mmlu_security_studies.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_sociology.yaml
...ve_worlds/output_variation/style_03/a/mmlu_sociology.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_us_foreign_policy.yaml
...s/output_variation/style_03/a/mmlu_us_foreign_policy.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_virology.yaml
...ive_worlds/output_variation/style_03/a/mmlu_virology.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_world_religions.yaml
...lds/output_variation/style_03/a/mmlu_world_religions.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/_mmlu.yaml
...alternative_worlds/output_variation/style_03/b/_mmlu.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/_template_yaml
...rnative_worlds/output_variation/style_03/b/_template_yaml
+0
-11
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_abstract_algebra.yaml
...ds/output_variation/style_03/b/mmlu_abstract_algebra.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_anatomy.yaml
...tive_worlds/output_variation/style_03/b/mmlu_anatomy.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_astronomy.yaml
...ve_worlds/output_variation/style_03/b/mmlu_astronomy.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_business_ethics.yaml
...lds/output_variation/style_03/b/mmlu_business_ethics.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_clinical_knowledge.yaml
.../output_variation/style_03/b/mmlu_clinical_knowledge.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_biology.yaml
...lds/output_variation/style_03/b/mmlu_college_biology.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_chemistry.yaml
...s/output_variation/style_03/b/mmlu_college_chemistry.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_computer_science.yaml
...t_variation/style_03/b/mmlu_college_computer_science.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_mathematics.yaml
...output_variation/style_03/b/mmlu_college_mathematics.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_medicine.yaml
...ds/output_variation/style_03/b/mmlu_college_medicine.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_physics.yaml
...lds/output_variation/style_03/b/mmlu_college_physics.yaml
+0
-6
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_computer_security.yaml
...s/output_variation/style_03/b/mmlu_computer_security.yaml
+0
-6
No files found.
Too many changes to show.
To preserve performance only
1000 of 1000+
files are displayed.
Plain diff
Email patch
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_public_relations.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
public_relations"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
public
\
\
relations.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03a_social_sciences"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_public_relations"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_security_studies.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
security_studies"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
security
\
\
studies.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03a_social_sciences"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_security_studies"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_sociology.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
sociology"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
sociology.
\n\
\n
"
"
group"
:
"
mmlu_alt_ov_03a_social_sciences"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_sociology"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_us_foreign_policy.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
us_foreign_policy"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
us
\
\
foreign
policy.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03a_social_sciences"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_us_foreign_policy"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_virology.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
virology"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
virology.
\n\
\n
"
"
group"
:
"
mmlu_alt_ov_03a_other"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_virology"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/a/mmlu_world_religions.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
world_religions"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
world
\
\
religions.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03a_humanities"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03a_world_religions"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/_mmlu.yaml
deleted
100644 → 0
View file @
3e5e9da2
group
:
mmlu_alt_ov_03b
task
:
-
mmlu_alt_ov_03b_stem
-
mmlu_alt_ov_03b_other
-
mmlu_alt_ov_03b_social_sciences
-
mmlu_alt_ov_03b_humanities
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/_template_yaml
deleted
100644 → 0
View file @
3e5e9da2
dataset_path: hails/mmlu_no_train
test_split: test
fewshot_split: dev
output_type: multiple_choice
doc_to_text: !function ../../../styles.template_03
doc_to_choice: !function ../../../styles.choice_03b
doc_to_target: answer
metric_list:
- metric: acc
- metric: acc_norm
- metric: brier_score
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_abstract_algebra.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
abstract_algebra"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
abstract
\
\
algebra.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_abstract_algebra"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_anatomy.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
anatomy"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
anatomy.
\n\
\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_anatomy"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_astronomy.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
astronomy"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
astronomy.
\n\
\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_astronomy"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_business_ethics.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
business_ethics"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
business
\
\
ethics.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_other"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_business_ethics"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_clinical_knowledge.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
clinical_knowledge"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
clinical
\
\
knowledge.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_other"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_clinical_knowledge"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_biology.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_biology"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
biology.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_biology"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_chemistry.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_chemistry"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
chemistry.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_chemistry"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_computer_science.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_computer_science"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
computer
science.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_computer_science"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_mathematics.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_mathematics"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
mathematics.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_mathematics"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_medicine.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_medicine"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
medicine.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_other"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_medicine"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_college_physics.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
college_physics"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
college
\
\
physics.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_college_physics"
lm_eval/tasks/mmlu/alternative_worlds/output_variation/style_03/b/mmlu_computer_security.yaml
deleted
100644 → 0
View file @
3e5e9da2
"
dataset_name"
:
"
computer_security"
"
description"
:
"
The
following
are
multiple
choice
questions
(with
answers)
about
computer
\
\
security.
\n\n
"
"
group"
:
"
mmlu_alt_ov_03b_stem"
"
include"
:
"
_template_yaml"
"
task"
:
"
mmlu_alt_ov_03b_computer_security"
Prev
1
…
31
32
33
34
35
36
37
38
39
…
50
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment