Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
f23ae748
Commit
f23ae748
authored
Sep 03, 2023
by
lintangsutawika
Browse files
add mmlu variants
parent
191458b8
Changes
235
Hide whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
158 additions
and
0 deletions
+158
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_astronomy.yaml
.../tasks/mmlu/flan_n_shot/mmlu_loglikelihood_astronomy.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_business_ethics.yaml
.../mmlu/flan_n_shot/mmlu_loglikelihood_business_ethics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_clinical_knowledge.yaml
...lu/flan_n_shot/mmlu_loglikelihood_clinical_knowledge.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_biology.yaml
.../mmlu/flan_n_shot/mmlu_loglikelihood_college_biology.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_chemistry.yaml
...mlu/flan_n_shot/mmlu_loglikelihood_college_chemistry.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_computer_science.yaml
...n_n_shot/mmlu_loglikelihood_college_computer_science.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_mathematics.yaml
...u/flan_n_shot/mmlu_loglikelihood_college_mathematics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_medicine.yaml
...mmlu/flan_n_shot/mmlu_loglikelihood_college_medicine.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_physics.yaml
.../mmlu/flan_n_shot/mmlu_loglikelihood_college_physics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_computer_security.yaml
...mlu/flan_n_shot/mmlu_loglikelihood_computer_security.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_conceptual_physics.yaml
...lu/flan_n_shot/mmlu_loglikelihood_conceptual_physics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_econometrics.yaml
...sks/mmlu/flan_n_shot/mmlu_loglikelihood_econometrics.yaml
+7
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_electrical_engineering.yaml
...lan_n_shot/mmlu_loglikelihood_electrical_engineering.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_elementary_mathematics.yaml
...lan_n_shot/mmlu_loglikelihood_elementary_mathematics.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_formal_logic.yaml
...sks/mmlu/flan_n_shot/mmlu_loglikelihood_formal_logic.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_global_facts.yaml
...sks/mmlu/flan_n_shot/mmlu_loglikelihood_global_facts.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_biology.yaml
...u/flan_n_shot/mmlu_loglikelihood_high_school_biology.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_chemistry.yaml
...flan_n_shot/mmlu_loglikelihood_high_school_chemistry.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_computer_science.yaml
...shot/mmlu_loglikelihood_high_school_computer_science.yaml
+8
-0
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_european_history.yaml
...shot/mmlu_loglikelihood_high_school_european_history.yaml
+8
-0
No files found.
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_astronomy.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
astronomy
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
astronomy.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_astronomy
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_business_ethics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
business_ethics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
business
ethics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_business_ethics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_clinical_knowledge.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
clinical_knowledge
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
clinical
knowledge.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_clinical_knowledge
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_biology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_biology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
biology.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_biology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_chemistry.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_chemistry
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
chemistry.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_chemistry
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_computer_science.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_computer_science
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
computer
science.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_computer_science
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_mathematics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_mathematics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
mathematics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_mathematics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_medicine.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_medicine
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
medicine.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_medicine
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_college_physics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
college_physics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
college
physics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_college_physics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_computer_security.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
computer_security
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
computer
security.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_computer_security
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_conceptual_physics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
conceptual_physics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
conceptual
physics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_conceptual_physics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_econometrics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
econometrics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
econometrics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_econometrics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_electrical_engineering.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
electrical_engineering
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
electrical
engineering.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_electrical_engineering
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_elementary_mathematics.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
elementary_mathematics
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
elementary
mathematics.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_elementary_mathematics
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_formal_logic.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
formal_logic
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
formal
logic.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_formal_logic
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_global_facts.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
global_facts
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
global
facts.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_global_facts
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_biology.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
high_school_biology
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
high
school
biology.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_high_school_biology
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_chemistry.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
high_school_chemistry
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
high
school
chemistry.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_high_school_chemistry
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_computer_science.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
high_school_computer_science
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
high
school
computer
science.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_high_school_computer_science
lm_eval/tasks/mmlu/flan_n_shot/mmlu_loglikelihood_high_school_european_history.yaml
0 → 100644
View file @
f23ae748
dataset_name
:
high_school_european_history
description
:
'
The
following
are
multiple
choice
questions
(with
answers)
about
high
school
european
history.
'
include
:
_mmlu_flan_loglikelihood_template_yaml
task
:
mmlu_flan_n_shot_loglikelihood_high_school_european_history
Prev
1
…
5
6
7
8
9
10
11
12
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment