Merge pull request #1060 from EleutherAI/fix-mmlu

[Refactor] Fix fewshot cot mmlu descriptions

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml

@@ -27,7 +27,7 @@
   \ (D) Initiating structure and considerations\nA: Let's think step by step. We refer\
   \ to Wikipedia articles on management for help. The Ohio State Leadership Studies\
   \ conducted in the 1940s identified initiating structure and consideration as the\
-  \ two main dimensions of leader behavior. The answer is (D)."
+  \ two main dimensions of leader behavior. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_management"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml

@@ -34,7 +34,7 @@
   \ and geographic variables.\n(A) Geodemographics (B) Product differentiation. (C)\
   \ ANSOFF matrix. (D) Brand management.\nA: Let's think step by step. We refer to\
   \ Wikipedia articles on marketing for help. Geodemographics is a natural outcome\
-  \ when combining demographic and geographic variables. The answer is (A)."
+  \ when combining demographic and geographic variables. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_marketing"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml

@@ -31,7 +31,7 @@
   \ think step by step. We refer to Wikipedia articles on medical genetics for help.\
   \ A Southern blot is a method in molecular biology for detecting specific DNA sequences\
   \ in a sample. Large triplet repeat expansions are usually detected with this method.\
-  \ The answer is (C)."
+  \ The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_medical_genetics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml

@@ -21,7 +21,7 @@
   \ the shortest NBA player in history to win an official slam dunk contest. The answer\
   \ is (A).\n\nQ: How many axles does a standard automobile have?\n(A) one (B) two\
   \ (C) four (D) eight\nA: Let's think step by step. We refer to Wikipedia for help.\
-  \ Most cars have two axles to rotate the wheels.. The answer is (B)."
+  \ Most cars have two axles to rotate the wheels.. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_miscellaneous"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml

@@ -43,7 +43,7 @@
   \ on ethical philosophy in South Africa. Metz has written the death penalty is unjustified\
   \ as it treats individuals as incapable of communal relations. It is unclear that\
   \ capital punishment is to the benefit of, or a deterrent of harm to the community.\
-  \ The answer is (A)."
+  \ The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_moral_disputes"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_scenarios.yaml

@@ -38,7 +38,7 @@
   \ house.\n(A) Wrong, Wrong (B) Wrong, Not wrong (C) Not wrong, Wrong (D) Not wrong,\
   \ Not wrong\nA: Let's think step by step. We refer to Wikipedia articles on moral\
   \ scenarios for help. Loving someone is not wrong. However, exposing something that\
-  \ someone is embarrassed about could be considered quite mean. The answer is (C)."
+  \ someone is embarrassed about could be considered quite mean. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_moral_scenarios"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml

@@ -42,7 +42,7 @@
   \ in disease risk between the two groups.\nA: Let's think step by step. We refer\
   \ to Wikipedia articles on nutrition for help. The risk ratio is not sufficiently\
   \ reduced that it could not be explained by random chance given the studies sample\
-  \ size. The answer is (C)."
+  \ size. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_nutrition"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml

@@ -24,7 +24,7 @@
   \ (D) none of the above.\nA: Let's think step by step. We refer to Wikipedia articles\
   \ on philosophy for help. Psychological egoism suggests that one behaves based on\
   \ what makes one feels good, hence it is a claim about human nature and how humans\
-  \ are capable of behaving. The answer is (C)."
+  \ are capable of behaving. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_philosophy"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml

@@ -36,7 +36,7 @@
   \ (C) frighten away enemies, in particular the Spaniards. (D) legitimize his kingship,\
   \ since his father was not royal.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on prehistory for help. Pacal built the temples as the funerary monument\
-  \ to legitimize his kingship. The answer is (D)."
+  \ to legitimize his kingship. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_prehistory"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml

@@ -42,7 +42,7 @@
   \ by step. We refer to Wikipedia articles on accounting for help. Among the four\
   \ transactions, only Proceeds from long-term debt belongs to the financing activities\
   \ section of cashflow, hence the amount reported should be $100000. The answer is\
-  \ (D)."
+  \ (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_professional_accounting"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml

@@ -100,7 +100,7 @@
   \ think step by step. We refer to Wikipedia articles on law for help. The Fourteenth\
   \ Amendment further supports the First Amendment by establishing a due process clause.\
   \ Hence the strongest argument should be the statute is overbroad and consequently\
-  \ invalid under the First and Fourteenth Amendments. The answer is (D)."
+  \ invalid under the First and Fourteenth Amendments. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_professional_law"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml

@@ -64,7 +64,7 @@
   A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
   \ The symptoms and the adrenal mass suggested pheochromocytoma, and the blood pressure\
   \ indicates hypertension. Phenoxybenzamine is used to treat hypertension caused\
-  \ by pheochromocytoma. The answer is (D)."
+  \ by pheochromocytoma. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_professional_medicine"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml

@@ -42,7 +42,7 @@
   A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
   \ Based on the circumstances, you should tell your client about the pros and cons\
   \ of each program, but it would be inappropriate to receive the bonus, so you should\
-  \ not claim the $50 bonus. The answer is (D)."
+  \ not claim the $50 bonus. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_professional_psychology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml

@@ -33,7 +33,7 @@
   \ think step by step. We refer to Wikipedia articles on public relations for help.\
   \ If a public relations media practitioner does not know the answer to a reporter's\
   \ question, they should say 'I don't know' and offer to provide the information\
-  \ later. The answer is (C)."
+  \ later. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_public_relations"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml

@@ -80,7 +80,7 @@
   \ conflict. It seeks to control by imposing compliance by removing any opportunity\
   \ for negotiation or concession.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on security studies for help. Coercive diplomacy uses the threat of force\
-  \ to induce the opponent to comply with demands. The answer is (B)."
+  \ to induce the opponent to comply with demands. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_security_studies"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml

@@ -37,7 +37,7 @@
   \ step. We refer to Wikipedia articles on sociology for help. The post-war welfare\
   \ state of 1948 aimed to provide free healthcare and education, full employment,\
   \ and universal welfare. But it did not aim to provide a minimum wage. The answer\
-  \ is (B)."
+  \ is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_sociology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml

@@ -34,7 +34,7 @@
   \ Obama (D) It reduced global use of the US dollar\nA: Let's think step by step.\
   \ We refer to Wikipedia articles on us foreign policy for help. The 2008 financial\
   \ crisis damanged the international reputation of the American model of political\
-  \ economy and capitalism. The answer is (A)."
+  \ economy and capitalism. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_us_foreign_policy"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml

@@ -25,7 +25,7 @@
   \ a helper virus (C) Only replicate in dividing cells (D) Can integrate into host\
   \ chromosomes\nA: Let's think step by step. We refer to Wikipedia articles on virology\
   \ for help. Paroviruses are highly impactful because they do not have nucleic acid.\
-  \ The answer is (A)."
+  \ The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_virology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml

@@ -21,7 +21,7 @@
   \ of the covenant for Jewish males?\n(A) The rainbow (B) Circumcision (C) A son\
   \ (D) Bar mitzvah\nA: Let's think step by step. We refer to Wikipedia articles on\
   \ world religions for help. In Judaism, the most distinctive sign of the covenant\
-  \ is circumcision (brit milah). The answer is (B)."
+  \ is circumcision (brit milah). The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_world_religions"