Merge branch 'main' into big-refactor_dp

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml

@@ -28,7 +28,7 @@
   \ of their nation or the world.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on global facts for help. As of 2019, most people tend to be optimistic\
   \ about their own future but pessimistic about the future of their nation or the\
-  \ world. The answer is (B)."
+  \ world. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_global_facts"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml

@@ -48,7 +48,7 @@
   \ proceed with cell division. Cues like these act by changing the activity of core\
   \ cell cycle regulators inside the cell. The most common regulators are cyclins\
   \ and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division.\
-  \ The answer is (D)."
+  \ The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_biology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml

@@ -44,7 +44,7 @@
   \ (aq) + H_{2}O \nightarrow H_{3}O^{+} + CH3COO^{-}$. The conjugate base is therefore\
   \ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\
   \ base. Therefore the maximum amount of acid that can be added will be equal to\
-  \ the amount of acetate ion, or 2 moles. The answer is (C)."
+  \ the amount of acetate ion, or 2 moles. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_chemistry"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml

@@ -65,7 +65,7 @@
   \ Choice C is incorrect because it incorrectly increments the variable count until\
   \ its value is greater than 100, regardless of the elements in the list. Choice\
   \ D is incorrect because its step 3 does not increment the value of position, so\
-  \ it will repeat forever. The answer is (B)."
+  \ it will repeat forever. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_computer_science"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml

@@ -163,7 +163,7 @@
   \ think step by step. We refer to Wikipedia articles on european history for help.\
   \ Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\
   \ against the monoplization of power and advocated for a system of checks and balances\
-  \ in government to prevent the rise of despotism. The answer is (B)."
+  \ in government to prevent the rise of despotism. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_european_history"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml

@@ -31,7 +31,7 @@
   \ the crude death rate. (C) doubling time from the crude birth rate. (D) fertility\
   \ rate from the crude death rate.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on geography for help. The difference between number of births and deaths\
-  \ gives the population increase at any given time. The answer is (A)."
+  \ gives the population increase at any given time. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_geography"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml

@@ -37,7 +37,7 @@
   \ have greatly increased presidential powers\nA: Let's think step by step. We refer\
   \ to Wikipedia articles on government and politics for help. The US Constitution\
   \ is not very specific about the powers of the president, leading to uncertainty\
-  \ over its limits. The answer is (A)."
+  \ over its limits. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_government_and_politics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml

@@ -31,7 +31,7 @@
   \ dozens of American cities.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on macroeconomics for help. The economic transactions related to the\
   \ performance of the American pop-singer in Paris happens entirely outside the U.S.\
-  \ and hence is not included in the GDP numbers. The answer is (C)."
+  \ and hence is not included in the GDP numbers. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_macroeconomics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml

@@ -31,7 +31,7 @@
   \ (Assume that all three lights blink simultaneously at the very beginning of the\
   \ dance.)\n(A) 3 (B) 15 (C) 6 (D) 5\nA: Let's think step by step. The least common\
   \ multiple of 2, 3 and 5 is 30, so during a 7 minute dance, all the three lights\
-  \ will come on at the same time $2*7+1=15$ times. The answer is (B)."
+  \ will come on at the same time $2*7+1=15$ times. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_mathematics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml

@@ -34,7 +34,7 @@
   \ effect.\nA: Let's think step by step. We refer to Wikipedia articles on microeconomics\
   \ for help. An increase in the construction of new houses means an increase demand\
   \ of in-house painting, thus increases the demand for housepainters. The answer\
-  \ is (C)."
+  \ is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_microeconomics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml

@@ -33,7 +33,7 @@
   A: Let's think step by step. At the closed end of the pipe, the particles cannot\
   \ have any net displacement because the pipe closure stops them. So the particle\
   \ displacement is at a node. This closure also causes the pressure to be maximal,\
-  \ i.e. an antinode. The answer is (B)."
+  \ i.e. an antinode. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_physics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml

@@ -42,7 +42,7 @@
   \ internal locus of control.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on psychology for help. People with an external locus of control believes\
   \ fate and luck play an important role in their lives, while people with an internal\
-  \ locus of control believes they control their lives. The answer is (D)."
+  \ locus of control believes they control their lives. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_psychology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml

@@ -52,7 +52,7 @@
   \ can occur even when the two variables are not perfectly correlated. Statement\
   \ B is false because uncorrelated variables regression lines can have slope zero.\
   \ Statement C is false because correlation is symmetric in the two random variables.\
-  \ The answer is (D)."
+  \ The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_statistics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml

@@ -128,7 +128,7 @@
   \ on us history for help. Anti-Federalists do not believe centralized government\
   \ power, and suspect Washington's military response to Whiskey Rebellion. Bacon's\
   \ Rebellion and Pontiac's Rebellion happen before the Revolution and they can be\
-  \ ruled out. The answer is (C)."
+  \ ruled out. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_us_history"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml

@@ -76,7 +76,7 @@
   \ most likely belong?\n(A) Hinduism (B) Buddhism (C) Shintoism (D) Zoroastrianism\n\
   A: Let's think step by step. We refer to Wikipedia articles on world history for\
   \ help. Brahman refers to the ultimate reality of all things in the Hindu religion.\
-  \ In contrast, Buddhism does not have a concept of supreme God. The answer is (A)."
+  \ In contrast, Buddhism does not have a concept of supreme God. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_high_school_world_history"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml

@@ -22,7 +22,7 @@
   \ likely destination?\n(A) Texas (B) California (C) Hawaii (D) Vermont\nA: Let's\
   \ think step by step. We refer to Wikipedia articles on human aging for help. Texas\
   \ does not have state tax, and has low cost of living compared with the other three\
-  \ options. The answer is (A)."
+  \ options. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_human_aging"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml

@@ -28,7 +28,7 @@
   \ during the third trimester (D) all through the pregnancy\nA: Let's think step\
   \ by step. We refer to Wikipedia articles on human sexuality for help. Morning sickness\
   \ usually begins by nine weeks after conception, corresponding to the first trimester.\
-  \ The answer is (A)."
+  \ The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_human_sexuality"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml

@@ -48,7 +48,7 @@
   \ affairs of States (D) Article 2(4) encompasses force directed only against a State's\
   \ territorial integrity\nA: Let's think step by step. We refer to Wikipedia articles\
   \ on international law for help. Article 2(4) of the UN Charter prohibits states\
-  \ from using armed forces in their international relations. The answer is (A)."
+  \ from using armed forces in their international relations. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_international_law"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml

@@ -39,7 +39,7 @@
   \ help. The POP would opt for the 'maximin' strategy, opt for the 'difference principle',\
   \ and reject the 'system of natural liberty', but the POP would not choose equality\
   \ above liberty, since the POP assume both equal and free citizens. The answer is\
-  \ (A)."
+  \ (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_jurisprudence"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml

@@ -39,7 +39,7 @@
   A: Let's think step by step. We refer to Wikipedia articles on logical fallacies\
   \ for help. The argument against the person fallacy occurs when someone irrelevantly\
   \ attacks the character of an opposing arguer, instead of addressing that opponent's\
-  \ arguments. The answer is (C)."
+  \ arguments. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_logical_fallacies"