Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
f23ae748
Commit
f23ae748
authored
Sep 03, 2023
by
lintangsutawika
Browse files
add mmlu variants
parent
191458b8
Changes
235
Hide whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
152 additions
and
0 deletions
+152
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_moral_disputes.yaml
...s/mmlu/flan_n_shot/mmlu_loglikelihood_moral_disputes.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_moral_scenarios.yaml
.../mmlu/flan_n_shot/mmlu_loglikelihood_moral_scenarios.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_nutrition.yaml
.../tasks/mmlu/flan_n_shot/mmlu_loglikelihood_nutrition.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_philosophy.yaml
...tasks/mmlu/flan_n_shot/mmlu_loglikelihood_philosophy.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_prehistory.yaml
...tasks/mmlu/flan_n_shot/mmlu_loglikelihood_prehistory.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_accounting.yaml
...an_n_shot/mmlu_loglikelihood_professional_accounting.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_law.yaml
...mmlu/flan_n_shot/mmlu_loglikelihood_professional_law.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_medicine.yaml
...flan_n_shot/mmlu_loglikelihood_professional_medicine.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_psychology.yaml
...an_n_shot/mmlu_loglikelihood_professional_psychology.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_public_relations.yaml
...mmlu/flan_n_shot/mmlu_loglikelihood_public_relations.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_security_studies.yaml
...mmlu/flan_n_shot/mmlu_loglikelihood_security_studies.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_sociology.yaml
.../tasks/mmlu/flan_n_shot/mmlu_loglikelihood_sociology.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_us_foreign_policy.yaml
...mlu/flan_n_shot/mmlu_loglikelihood_us_foreign_policy.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_virology.yaml
...l/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_virology.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_world_religions.yaml
.../mmlu/flan_n_shot/mmlu_loglikelihood_world_religions.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_machine_learning.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_machine_learning.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_management.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_management.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_marketing.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_marketing.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_medical_genetics.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_medical_genetics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_miscellaneous.yaml
lm_eval/tasks/mmlu/flan_n_shot/mmlu_miscellaneous.yaml
+7
-0
No files found.
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_moral_disputes.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
moral_disputes
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
moral
disputes.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_moral_disputes
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_moral_scenarios.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
moral_scenarios
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
moral
scenarios.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_moral_scenarios
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_nutrition.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
nutrition
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
nutrition.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_nutrition
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_philosophy.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
philosophy
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
philosophy.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_philosophy
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_prehistory.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
prehistory
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
prehistory.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_prehistory
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_accounting.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_accounting
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
accounting.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_professional_accounting
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_law.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_law
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
law.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_professional_law
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_medicine.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_medicine
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
medicine.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_professional_medicine
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_professional_psychology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
professional_psychology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
professional
psychology.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_professional_psychology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_public_relations.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
public_relations
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
public
relations.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_public_relations
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_security_studies.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
security_studies
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
security
studies.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_security_studies
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_sociology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
sociology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
sociology.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_sociology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_us_foreign_policy.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
us_foreign_policy
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
us
foreign
policy.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_us_foreign_policy
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_virology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
virology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
virology.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_virology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_world_religions.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
world_religions
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
world
religions.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_world_religions
lm_eval/tasks/mmlu/flan_n_shot/mmlu_machine_learning.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
machine_learning
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
machine
learning.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_machine_learning
lm_eval/tasks/mmlu/flan_n_shot/mmlu_management.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
management
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
management.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_management
lm_eval/tasks/mmlu/flan_n_shot/mmlu_marketing.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
marketing
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
marketing.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_marketing
lm_eval/tasks/mmlu/flan_n_shot/mmlu_medical_genetics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
medical_genetics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
medical
genetics.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_medical_genetics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_miscellaneous.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
miscellaneous
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
miscellaneous.
'
include
:
_mmlu_flan_generative_template_yaml
task
:
mmlu_flan_n_shot_miscellaneous
Prev
1
…
7
8
9
10
11
12
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment