Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
f23ae748
Commit
f23ae748
authored
Sep 03, 2023
by
lintangsutawika
Browse files
add mmlu variants
parent
191458b8
Changes
235
Hide whitespace changes
Inline
Side-by-side
Showing
15 changed files
with
115 additions
and
0 deletions
+115
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_disputes.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_disputes.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_scenarios.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_scenarios.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_nutrition.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_nutrition.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_philosophy.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_philosophy.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_prehistory.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_prehistory.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_accounting.yaml
.../tasks/mmlu/flan_n_shot/mmlu_professional_accounting.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_law.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_law.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_medicine.yaml
...al/tasks/mmlu/flan_n_shot/mmlu_professional_medicine.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_psychology.yaml
.../tasks/mmlu/flan_n_shot/mmlu_professional_psychology.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_public_relations.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_public_relations.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_security_studies.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_security_studies.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_sociology.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_sociology.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_us_foreign_policy.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_us_foreign_policy.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_virology.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_virology.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_world_religions.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_world_religions.yaml
+8
-0
No files found.
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_disputes.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
moral_disputes
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
moral
disputes.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_moral_disputes
lm_eval/tasks/mmlu/flan_n_shot/mmlu_moral_scenarios.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
moral_scenarios
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
moral
scenarios.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_moral_scenarios
lm_eval/tasks/mmlu/flan_n_shot/mmlu_nutrition.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
nutrition
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
nutrition.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_nutrition
lm_eval/tasks/mmlu/flan_n_shot/mmlu_philosophy.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
philosophy
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
philosophy.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_philosophy
lm_eval/tasks/mmlu/flan_n_shot/mmlu_prehistory.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
prehistory
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
prehistory.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_prehistory
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_accounting.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_accounting
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
accounting.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_professional_accounting
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_law.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_law
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
law.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_professional_law
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_medicine.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_medicine
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
medicine.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_professional_medicine
lm_eval/tasks/mmlu/flan_n_shot/mmlu_professional_psychology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_psychology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
psychology.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_professional_psychology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_public_relations.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
public_relations
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
public
relations.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_public_relations
lm_eval/tasks/mmlu/flan_n_shot/mmlu_security_studies.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
security_studies
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
security
studies.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_security_studies
lm_eval/tasks/mmlu/flan_n_shot/mmlu_sociology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
sociology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
sociology.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_sociology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_us_foreign_policy.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
us_foreign_policy
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
us
foreign
policy.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_us_foreign_policy
lm_eval/tasks/mmlu/flan_n_shot/mmlu_virology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
virology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
virology.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_virology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_world_religions.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
world_religions
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
world
religions.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_world_religions
Prev
1
…
8
9
10
11
12
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment