Merge branch 'main' into multimodal-prototyping

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml

@@ -65,6 +65,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on international
       law for help. Article 2(4) of the UN Charter prohibits states from using armed
       forces in their international relations. The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_international_law

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml

@@ -54,6 +54,6 @@ fewshot_config:
       principle'', and reject the ''system of natural liberty'', but the POP would
       not choose equality above liberty, since the POP assume both equal and free
       citizens. The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_jurisprudence

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml

@@ -56,6 +56,6 @@ fewshot_config:
       fallacies for help. The argument against the person fallacy occurs when someone
       irrelevantly attacks the character of an opposing arguer, instead of addressing
       that opponent''s arguments. The answer is (C).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_logical_fallacies

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml

@@ -69,6 +69,6 @@ fewshot_config:
       \ epsilon when we have N samples if 2 exp(-2 epsilon^2 N)<0.05, this implies\
       \ that N > -1/(2*epsilon**2) log ( 0.05/2 )= log (40)*5000. Since log(40)>1,\
       \ we have that one needs more than 1000 examples. The answer is (D).\n\n"
-group: mmlu_flan_cot_fewshot_stem
+tag: mmlu_flan_cot_fewshot_stem
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_machine_learning

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml

@@ -41,6 +41,6 @@ fewshot_config:
       for help. The Ohio State Leadership Studies conducted in the 1940s identified
       initiating structure and consideration as the two main dimensions of leader
       behavior. The answer is (D).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_management

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml

@@ -51,6 +51,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on marketing
       for help. Geodemographics is a natural outcome when combining demographic and
       geographic variables. The answer is (A).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_marketing

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml

@@ -46,6 +46,6 @@ fewshot_config:
       genetics for help. A Southern blot is a method in molecular biology for detecting
       specific DNA sequences in a sample. Large triplet repeat expansions are usually
       detected with this method. The answer is (C).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_medical_genetics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml

@@ -38,6 +38,6 @@ fewshot_config:
       (A) one (B) two (C) four (D) eight'
     target: 'Let''s think step by step. We refer to Wikipedia for help. Most cars
       have two axles to rotate the wheels.. The answer is (B).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_miscellaneous

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml

@@ -59,6 +59,6 @@ fewshot_config:
       as it treats individuals as incapable of communal relations. It is unclear that
       capital punishment is to the benefit of, or a deterrent of harm to the community.
       The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_moral_disputes

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml

@@ -57,6 +57,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on moral scenarios
       for help. Loving someone is not wrong. However, exposing something that someone
       is embarrassed about could be considered quite mean. The answer is (C).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_moral_scenarios

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml

@@ -58,6 +58,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on nutrition
       for help. The risk ratio is not sufficiently reduced that it could not be explained
       by random chance given the studies sample size. The answer is (C).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_nutrition

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml

@@ -39,6 +39,6 @@ fewshot_config:
       for help. Psychological egoism suggests that one behaves based on what makes
       one feels good, hence it is a claim about human nature and how humans are capable
       of behaving. The answer is (C).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_philosophy

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml

@@ -54,6 +54,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on prehistory
       for help. Pacal built the temples as the funerary monument to legitimize his
       kingship. The answer is (D).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_prehistory

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml

@@ -58,6 +58,6 @@ fewshot_config:
       for help. Among the four transactions, only Proceeds from long-term debt belongs
       to the financing activities section of cashflow, hence the amount reported should
       be $100000. The answer is (D).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_professional_accounting

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml

@@ -117,6 +117,6 @@ fewshot_config:
       a due process clause. Hence the strongest argument should be the statute is
       overbroad and consequently invalid under the First and Fourteenth Amendments.
       The answer is (D).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_professional_law

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml

@@ -77,6 +77,6 @@ fewshot_config:
       for help. The symptoms and the adrenal mass suggested pheochromocytoma, and
       the blood pressure indicates hypertension. Phenoxybenzamine is used to treat
       hypertension caused by pheochromocytoma. The answer is (D).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_professional_medicine

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml

@@ -57,6 +57,6 @@ fewshot_config:
       for help. Based on the circumstances, you should tell your client about the
       pros and cons of each program, but it would be inappropriate to receive the
       bonus, so you should not claim the $50 bonus. The answer is (D).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_professional_psychology

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml

@@ -50,6 +50,6 @@ fewshot_config:
       for help. If a public relations media practitioner does not know the answer
       to a reporter''s question, they should say ''I don''t know'' and offer to provide
       the information later. The answer is (C).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_public_relations

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml

@@ -99,6 +99,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on security
       studies for help. Coercive diplomacy uses the threat of force to induce the
       opponent to comply with demands. The answer is (B).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_security_studies

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml

@@ -53,6 +53,6 @@ fewshot_config:
       for help. The post-war welfare state of 1948 aimed to provide free healthcare
       and education, full employment, and universal welfare. But it did not aim to
       provide a minimum wage. The answer is (B).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_sociology