mmlu_pro:

-changed task 'other' to 'miscellaneous'
there is already a group named 'other'
task and group with the same alias (e.g. mmlu_pro_other_generative) throws an error

-fixed yaml backslash escape for fewshot cot

lm_eval/tasks/mmlu_pro/continuation/mmlu_pro_other.yaml → lm_eval/tasks/mmlu_pro/continuation/mmlu_pro_miscellaneous.yaml

@@ -3,4 +3,4 @@
   \n"
 "group": "mmlu_continuation_other"
 "include": "_continuation_template_yaml"
-"task": "mmlu_continuation_other"
+"task": "mmlu_continuation_miscellaneous"

lm_eval/tasks/mmlu_pro/default/mmlu_pro_other.yaml → lm_eval/tasks/mmlu_pro/default/mmlu_pro_miscellaneous.yaml

@@ -4,5 +4,5 @@
 "group": "mmlu_pro_other"
 "group_alias": "other"
 "include": "_default_template_yaml"
-"task": "mmlu_pro_other"
-"task_alias": "other"
+"task": "mmlu_pro_miscellaneous"
+"task_alias": "miscellaneous"

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_anatomy.yaml

-dataset_name: anatomy
-description: The following are multiple choice questions (with answers) about anatomy.
-fewshot_config:
-  sampler: first_n
-  samples:
-  - question: 'Which of the following is the body cavity that contains the pituitary
-      gland?
-
-      (A) Abdominal (B) Cranial (C) Pleural (D) Spinal'
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for\
-      \ help. Let\u2019s solve this problem step by step. The pituitary gland is the\
-      \ major endocrine gland attached to the base of the brain, and it is contained\
-      \ in the Cranial cavity. The answer is (B)."
-  - question: 'Which of these branches of the trigeminal nerve contain somatic motor
-      processes?
-
-      (A) The supraorbital nerve (B) The infraorbital nerve (C) The mental nerve (D)
-      None of the above'
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for\
-      \ help. Let\u2019s solve this problem step by step. \nWe know the following:\
-      \ (A) The supraorbital nerve (also known as the frontal nerve) is the largest\
-      \ branch of the ophthalmic nerve and branch of ophthalmic division of the trigeminal\
-      \ nerve. (B) The infraorbital nerve is a branch of the maxillary division of\
-      \ the trigeminal nerve. (C) The mental nerve is a branch of the mandibular division\
-      \ of the trigeminal nerve. Because all these nerves are purely sensory nerves\
-      \ and do not contain any somatic motor processes. Therefore, the answer should\
-      \ be none of the above, which is (D). The answer is (D)."
-  - question: 'In Angle''s Class II Div 2 occlusion there is
-
-      (A) excess overbite of the upper lateral incisors. (B) negative overjet of the
-      upper central incisors. (C) excess overjet of the upper lateral incisors. (D)
-      excess overjet of the upper central incisors.'
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for\
-      \ help. Let\u2019s solve this problem step by step. This is a question related\
-      \ to anatomy and orthodontics. Excess overjet is associated with Class II occlusions;\
-      \ therefore, we can safely eliminate (B) from the list, as negative overjet\
-      \ is often associated with Class III occlusions. Now, we need to determine the\
-      \ location of the excess overjet, and that would be the upper (maxillary) lateral\
-      \ incisors. Only (C) has the correct information. The answer is (C)."
-  - question: 'The pleura
-
-      (A) have no sensory innervation. (B) are separated by a 2 mm space. (C) extend
-      into the neck. (D) are composed of respiratory epithelium.'
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for\
-      \ help. Let\u2019s solve this problem step by step. First, recall that the pleura\
-      \ refers to the thin layer of tissue that covers the lungs and lines the interior\
-      \ wall of the chest cavity. Now, let\u2019s look at each option:\nOption (A):\
-      \ \u201CThe pleura have no sensory innervation.\u201D This information is not\
-      \ correct. The pleura do have a sensory innervation.\nOption (B): \u201CThe\
-      \ pleura are separated by a 2 mm space.\u201D This information is not correct.\
-      \ There is a very thin \u201Cpotential\u201D space between the layers of the\
-      \ pleura; however, it is typically filled with serous pleural fluid. \nOption\
-      \ (C): \u201CThe pleura extend into the neck.\u201D This information is actuakky\
-      \ true. The cervical pleura, also known as the dome of the pleuradome of the\
-      \ pleura, lines the extendsiton of the pleural cavity into the neck.\nOption\
-      \ (D): \u201CThe pleura are composed of respiratory epithelium.\u201D This information\
-      \ is not correct. The pleaura are composed of connective tissue (CT).\nBecause\
-      \ (A), (B), and (D) are all incorrect, (D) is the only correct answer. The answer\
-      \ is (C)."
-  - question: 'What is the embryological origin of the hyoid bone?
-
-      (A) The first pharyngeal arch (B) The first and second pharyngeal arches (C)
-      The second pharyngeal arch (D) The second and third pharyngeal arches'
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for\
-      \ help. Let\u2019s solve this problem step by step. The hyoid bone, which is\
-      \ also known as the hyooid, is a a small U-shaped bone located in the anterior\
-      \ neck. In its resting position, it lies between the ase of the mandible and\
-      \ the third cervical vertebrae. We know that the second and the third pharyngeal\
-      \ arches give rise to the horns of the hyoid bone; therefore, the embryological\
-      \ origin of the hyoid bone are the second and the third pharyngeal arches\u2014\
-      this information is covered in the last option (D). Therefore, we conclude that\
-      \ (D) must be the correct answer. The answer is (D).\n\n"
-group: mmlu_flan_cot_fewshot_stem
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_anatomy

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_astronomy.yaml

-dataset_name: astronomy
-description: The following are multiple choice questions (with answers) about astronomy.
-fewshot_config:
-  sampler: first_n
-  samples:
-  - question: 'Where do most short-period comets come from and how do we know?
-
-      (A) The Kuiper belt; short period comets tend to be in the plane of the solar
-      system just like the Kuiper belt. (B) The Kuiper belt; short period comets tend
-      to come from random directions indicating a spherical distribution of comets
-      called the Kuiper belt. (C) The asteroid belt; short period comets have orbital
-      periods similar to asteroids like Vesta and are found in the plane of the solar
-      system just like the asteroid belt. (D) The Oort cloud; short period comets
-      tend to be in the plane of the solar system just like the Oort cloud.'
-    target: Let's think step by step. Most short-period comets come from the Kuiper
-      belt, and we know because short period coments tend to be in the plane of the
-      solar system, just like the Kuiper belt is. The answer is (A).
-  - question: 'You are pushing a truck along a road. Would it be easier to accelerate
-      this truck on Mars? Why? (Assume there is no friction)
-
-      (A) It would be harder since the truck is heavier on Mars. (B) It would be easier
-      since the truck is lighter on Mars. (C) It would be harder since the truck is
-      lighter on Mars. (D) It would be the same no matter where you are.'
-    target: "Let's think step by step. If we assume that there is no friction, the\
-      \ force needed to accelerate the truck is by Newton\u2019s second law only dependent\
-      \ on the mass of the truck. Hence (A), (B) and (C) are incorrect since it doesn\u2019\
-      t matter that it\u2019s on Mars, and (D) is the correct answer. The answer is\
-      \ (D)."
-  - question: 'Say the pupil of your eye has a diameter of 5 mm and you have a telescope
-      with an aperture of 50 cm. How much more light can the telescope gather than
-      your eye?
-
-      (A) 10000 times more (B) 100 times more (C) 1000 times more (D) 10 times more'
-    target: Let's think step by step. The amount of light is proportional to the aperture
-      area $A = \pi D^2/4$ for a lens with diameter $D$, so the relative amounts of
-      light between the eye with diameter 5mm and the telescope with diameter 50mm
-      is $(50 cm)^2/(5mm)^2 = 10000$. The answer is (A).
-  - question: 'Why isn''t there a planet where the asteroid belt is located?
-
-      (A) A planet once formed here but it was broken apart by a catastrophic collision.
-      (B) There was not enough material in this part of the solar nebula to form a
-      planet. (C) There was too much rocky material to form a terrestrial planet but
-      not enough gaseous material to form a jovian planet. (D) Resonance with Jupiter
-      prevented material from collecting together to form a planet.'
-    target: "Let's think step by step. The asteroid belt is a stellar disc consisting\
-      \ of a large number of asteroids between Mars and Jupiter's orbits. The asteroids\
-      \ in this belt are affected by the gravitational pull from both other asteroids\
-      \ and nearby planets. Due to the strong gravitational force of Jupiter there\
-      \ are resonances that give rise to low density regions of asteroids known as\
-      \ the Kirkwood gap. So (B) and (C) are not correct since it\u2019s not a lack\
-      \ of material that prevents a planet from being formed, and (A) is incorrect\
-      \ because the Kirkwood gap would have prevented a planet from forming in the\
-      \ first place, and (D) is the correct option. The answer is (D)."
-  - question: 'Why is Mars red?
-
-      (A) Because the surface is covered with heavily oxidized ("rusted") minerals.
-      (B) Because the atmosphere scatters more light at bluer wavelengths transmitting
-      mostly red light. (C) Because Mars is covered with ancient lava flows which
-      are red in color. (D) Because flowing water on Mars''s surface altered the surface
-      minerals several billion years ago.'
-    target: 'Let''s think step by step. Option (B) is not correct because if the red
-      color was caused by the scattering off the atmosphere, then the earth with a
-      much thicker atmosphere would also look red. Options (C) and (D) are not specific
-      enough about why the color of the surface would be red, while (A) is correct
-      because it explains that the surface is red due to the rusted materials on the
-      surface and the red color comes from the rust. So the correct option is (A).
-      The answer is (A).'
-group: mmlu_flan_cot_fewshot_stem
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_astronomy

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_biology.yaml

@@ -3,21 +3,16 @@ description: The following are multiple choice questions (with answers) about bi
 fewshot_config:
   sampler: first_n
   samples:
-  - question: "Which of the following represents an accurate statement concerning arthropods?
-      (A) They possess an exoskeleton composed primarily of peptidoglycan. (B) They possess an open circulatory system with a dorsal heart. (C) They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources. (D) They lack paired, jointed appendages. (E) N/A (F) N/A (G) N/A (H) N/A (I) N/A (J) N/A"
-    target: "Let's think step by step. Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B), as arthropods have an open circulatory system with a dorsal tubular heart. The answer is (B)."
-  - question: "In a given population, 1 out of every 400 people has a cancer caused by a completely recessive allele, b. Assuming the population is in Hardy-Weinberg equilibrium, which of the following is the expected proportion of individuals who carry the b allele but are not expected to develop the cancer?
-      (A) 19/400 (B) 1/400 (C) 40/400 (D) 38/400 (E) 2/400 (F) 1/200 (G) 20/400 (H) 50/400 (I) N/A (J) N/A"
-    target: "Let's think step by step. According to the Hardy Weinberg Law, $p^2 + 2 p q + q^2 = 1$, and $p + q = 1$ where $p$ is the frequency of the dominant allele, $q$ is the frequency of the recessive allele, and $p^2$, $q^2$, and $2pq$ are the frequencies of dominant homozygous, recessive homozygous, and heterozygous individuals, respectively. ​The frequency of the recessive allele (q) is $\sqrt{rac{1}{400}} = 0.05$. We have $p = 1 - q = 0.95$. The frequency of heterozygous individuals is $2pq = 2 \cdot 0.05 \cdot 0.95 = 0.095$. The number of heterozygous individuals is equal to the frequency of heterozygous individuals times the size of the population, or $0.095 * 400 = 38$. So we end up with 38/400. The answer is (D)."
-  - question: "A mutation in a bacterial enzyme changed a previously polar amino acid into a nonpolar amino acid. This amino acid was located at a site distant from the enzyme’s active site. How might this mutation alter the enzyme’s substrate specificity?
-      (A) By changing the enzyme’s pH optimum (B) By changing the enzyme's molecular weight (C) An amino acid change away from the active site increases the enzyme's substrate specificity. (D) By changing the shape of the protein (E) By changing the enzyme's temperature optimum (F) By altering the enzyme's ability to be denatured (G) By changing the enzyme’s location in the cell (H) By changing the enzyme's color (I) An amino acid change away from the active site cannot alter the enzyme’s substrate specificity. (J) By altering the enzyme's rate of reaction"
-    target: "Let's think step by step. A change in an amino acid leads to a change in the primary structure of the protein. A change in the primary structure may lead to a change in the secondary and the tertiary structure of the protein. A change in the tertiary structure means a change in the shape of the protein, so (C) has to be correct. Since the change does not affect the active site of the enzyme, we do not expect the activity of the enzyme to be affected. The answer is (D)."
-  - question: "Which of the following is not a way to form recombinant DNA?
-      (A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation (E) N/A (F) N/A (G) N/A (H) N/A (I) N/A (J) N/A"
-    target: "Let's think step by step. The introduction of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique in molecular biology and scientific research. There are multiple ways foreign DNA can be introduced into cells including transformation, transduction, conjugation, and transfection. In contrast, (A) is not a way to form DNA: during translation the ribosomes synthesize proteins from RNA. The answer is (A)."
-  - question: "Which of the following is not known to be involved in the control of cell division?
-      (A) Microtubules (B) Checkpoints (C) DNA polymerase (D) Centrosomes (E) Cyclins (F) Mitochondria (G) Protein kinases (H) Fibroblast cells (I) N/A (J) N/A"
-    target: "Let's think step by step. Normal cells move through the cell cycle in a regulated way. At the checkpoint stage, they use information about their own internal state and cues from the environment around them to decide whether to proceed with cell division. Cues like these act by changing the activity of core cell cycle regulators inside the cell. The most common regulators are cyclins and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division. The answer is (H)."
+    - question: "Which of the following represents an accurate statement concerning arthropods? (A) They possess an exoskeleton composed primarily of peptidoglycan. (B) They possess an open circulatory system with a dorsal heart. (C) They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources. (D) They lack paired, jointed appendages. (E) N/A (F) N/A (G) N/A (H) N/A (I) N/A (J) N/A"
+      target: "Let's think step by step. Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B), as arthropods have an open circulatory system with a dorsal tubular heart. The answer is (B)."
+    - question: "In a given population, 1 out of every 400 people has a cancer caused by a completely recessive allele, b. Assuming the population is in Hardy-Weinberg equilibrium, which of the following is the expected proportion of individuals who carry the b allele but are not expected to develop the cancer? (A) 19/400 (B) 1/400 (C) 40/400 (D) 38/400 (E) 2/400 (F) 1/200 (G) 20/400 (H) 50/400 (I) N/A (J) N/A"
+      target: "Let's think step by step. According to the Hardy Weinberg Law, \\( p^2 + 2pq + q^2 = 1 \\), and \\( p + q = 1 \\) where \\( p \\) is the frequency of the dominant allele, \\( q \\) is the frequency of the recessive allele, and \\( p^2 \\), \\( q^2 \\), and \\( 2pq \\) are the frequencies of dominant homozygous, recessive homozygous, and heterozygous individuals, respectively. The frequency of the recessive allele (q) is \\( \\sqrt{\\frac{1}{400}} = 0.05 \\). We have \\( p = 1 - q = 0.95 \\). The frequency of heterozygous individuals is \\( 2pq = 2 \\cdot 0.05 \\cdot 0.95 = 0.095 \\). The number of heterozygous individuals is equal to the frequency of heterozygous individuals times the size of the population, or \\( 0.095 \\times 400 = 38 \\). So we end up with 38/400. The answer is (D)."
+    - question: "A mutation in a bacterial enzyme changed a previously polar amino acid into a nonpolar amino acid. This amino acid was located at a site distant from the enzyme’s active site. How might this mutation alter the enzyme’s substrate specificity? (A) By changing the enzyme’s pH optimum (B) By changing the enzyme's molecular weight (C) An amino acid change away from the active site increases the enzyme's substrate specificity. (D) By changing the shape of the protein (E) By changing the enzyme's temperature optimum (F) By altering the enzyme's ability to be denatured (G) By changing the enzyme’s location in the cell (H) By changing the enzyme's color (I) An amino acid change away from the active site cannot alter the enzyme’s substrate specificity. (J) By altering the enzyme's rate of reaction"
+      target: "Let's think step by step. A change in an amino acid leads to a change in the primary structure of the protein. A change in the primary structure may lead to a change in the secondary and the tertiary structure of the protein. A change in the tertiary structure means a change in the shape of the protein, so (C) has to be correct. Since the change does not affect the active site of the enzyme, we do not expect the activity of the enzyme to be affected. The answer is (D)."
+    - question: "Which of the following is not a way to form recombinant DNA? (A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation (E) N/A (F) N/A (G) N/A (H) N/A (I) N/A (J) N/A"
+      target: "Let's think step by step. The introduction of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique in molecular biology and scientific research. There are multiple ways foreign DNA can be introduced into cells including transformation, transduction, conjugation, and transfection. In contrast, (A) is not a way to form DNA: during translation the ribosomes synthesize proteins from RNA. The answer is (A)."
+    - question: "Which of the following is not known to be involved in the control of cell division? (A) Microtubules (B) Checkpoints (C) DNA polymerase (D) Centrosomes (E) Cyclins (F) Mitochondria (G) Protein kinases (H) Fibroblast cells (I) N/A (J) N/A"
+      target: "Let's think step by step. Normal cells move through the cell cycle in a regulated way. At the checkpoint stage, they use information about their own internal state and cues from the environment around them to decide whether to proceed with cell division. Cues like these act by changing the activity of core cell cycle regulators inside the cell. The most common regulators are cyclins and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division. The answer is (H)."
 group: mmlu_pro_flan_cot_fewshot_stem
 include: _mmlu_pro_flan_cot_fewshot_template_yaml
 task: mmlu_pro_flan_cot_fewshot_biology

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_chemistry.yaml

@@ -5,7 +5,7 @@ fewshot_config:
   samples:
   - question: "Predict the number of lines in the EPR spectrum of a solution of 13C-labelled methyl radical (13CH3•), assuming the lines do not overlap.
       (A) 10 (B) 8 (C) 4 (D) 20 (E) 12 (F) 3 (G) 16 (H) 5 (I) 24 (J) 6"
-    target: "Let's think step by step. The electron paramagnetic resonance spectrum will be split by two forms of interactions. The first is the hyperfine interaction with the 13C (nuclear spin $I = \frac{1}{2}$) which will split the spectrum into 2 lines. This will be further split into 4 lines by the interaction with three equivalent 1H nuclei. The total number of lines is therefore $2 \cdot 4 = 8$. The answer is (B)."
+    target: "Let's think step by step. The electron paramagnetic resonance spectrum will be split by two forms of interactions. The first is the hyperfine interaction with the 13C (nuclear spin $I = \\frac{1}{2}$) which will split the spectrum into 2 lines. This will be further split into 4 lines by the interaction with three equivalent 1H nuclei. The total number of lines is therefore $2 \\cdot 4 = 8$. The answer is (B)."
   - question: "Which of the following lists the hydrides of group-14 elements in order of thermal stability, from lowest to highest?
       (A) GeH4 < SnH4 < PbH4 < SiH4 < CH4 (B) SiH4 < GeH4 < SnH4 < PbH4 < CH4 (C) PbH4 < CH4 < SnH4 < GeH4 < SiH4 (D) PbH4 < SnH4 < CH4 < GeH4 < SiH4 (E) SnH4 < GeH4 < SiH4 < PbH4 < CH4 (F) CH4 < GeH4 < SnH4 < PbH4 < SiH4 (G) SiH4 < SnH4 < PbH4 < GeH4 < CH4 (H) CH4 < SiH4 < GeH4 < SnH4 < PbH4 (I) CH4 < PbH4 < GeH4 < SnH4 < SiH4 (J) PbH4 < SnH4 < GeH4 < SiH4 < CH4"
     target: "Let's think step by step. The thermal stability of group-14 hydrides decreases as we move from the top of group 14 to the bottom. The order of elements in the group from top to bottom is C, Si, Ge, Sn, Pb. Therefore in order of increasing thermal stability we have PbH4, SnH4, GeH4, SiH4, and CH4, or answer (J). The answer is (J)."
@@ -14,10 +14,10 @@ fewshot_config:
     target: "Let's think step by step. An acid anhydride is a compound that is derived by removing water from an acid. The chemical formula for water is H2O, which means that we need to determine which of these options, when combined with H2O, forms an acid. SO2, or Sulfur dioxide, when combined with H2O, makes H2SO4, or sulfuric acid. The answer is (E)."
   - question: "A new compound is synthesized and found to be a monoprotic acid with a molar mass of 248 g/mol. When 0.0050 mol of this acid are dissolved in 0.500 L of water, the pH is measured as 3.89. What is the pKa of this acid?
       (A) 5.78 (B) 4.78 (C) 4.56 (D) 6.89 (E) 7.78 (F) 3.89 (G) 1.23 (H) 2.89 (I) 2.33 (J) 5.33"
-    target: "Let's think step by step. Recall that $[A] = [H^{+}]$. Here, this is equal to $10^{-3.89}$. Then we have $K_{a} = \frac{[H^{+}][A^{-}]}{[HA]} = \frac{10^{-3.89} \cdot 10^{-3.89}}{10^{-2}}$. The resulting exponent is $-3.89 + (-3.89) - (-2) = 5.78$, therefore $K_a = 10^{-5.78}$. The $pK_a$ is the negative log of $K_a$, which is equal to $5.78$. The answer is (A)."
+    target: "Let's think step by step. Recall that $[A] = [H^{+}]$. Here, this is equal to $10^{-3.89}$. Then we have $K_{a} = \\frac{[H^{+}][A^{-}]}{[HA]} = \\frac{10^{-3.89} \\cdot 10^{-3.89}}{10^{-2}}$. The resulting exponent is $-3.89 + (-3.89) - (-2) = 5.78$, therefore $K_a = 10^{-5.78}$. The $pK_a$ is the negative log of $K_a$, which is equal to $5.78$. The answer is (A)."
   - question: "A solution contains 2.00 mole of acetic acid, CH3COOH, and 1.00 mole of calcium acetate, Ca(CH3COO)2. The solution is able to resist the addition of a small amount of strong acid or strong base with only minor changes in the pH of the solution. Larger quantities of strong acid or strong base can cause a significant change in pH. How many moles of nitric acid, HNO3, may be added before the pH begins to change significantly?
       (A) 0.250 mole (B) 0.500 mole (C) 3.00 mole (D) 1.00 mole (E) 3.50 mole (F) 1.50 mole (G) 2.50 mole (H) 4.00 mole (I) 0.750 mole (J) 2.00 mole"
-    target: "Let's think step by step. We would like to compute the buffer capacity of this solution. First we write the equation for the ionization of the weak acid, in this case of acetic acid. $CH_{3}COOH (aq) + H_{2}O \rightarrow H_{3}O^{+} + CH3COO^{-}$. The conjugate base is therefore the acetate ion. The added strong acid, Nitric acid, will react with the conjugate base. Therefore the maximum amount of acid that can be added will be equal to the amount of acetate ion, or 2 moles. The answer is (J)."
+    target: "Let's think step by step. We would like to compute the buffer capacity of this solution. First we write the equation for the ionization of the weak acid, in this case of acetic acid. $CH_{3}COOH (aq) + H_{2}O \\rightarrow H_{3}O^{+} + CH3COO^{-}$. The conjugate base is therefore the acetate ion. The added strong acid, Nitric acid, will react with the conjugate base. Therefore the maximum amount of acid that can be added will be equal to the amount of acetate ion, or 2 moles. The answer is (J)."
 group: mmlu_pro_flan_cot_fewshot_stem
 include: _mmlu_pro_flan_cot_fewshot_template_yaml
 task: mmlu_pro_flan_cot_fewshot_chemistry

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_engineering.yaml

@@ -5,16 +5,16 @@ fewshot_config:
   samples:
   - question: "A point pole has a strength of 4π * 10^-4 weber. The force in newtons on a point pole of 4π * 1.5 * 10^-4 weber placed at a distance of 10 cm from it will be
       (A) 25 N. (B) 12.5 N. (C) 15 N. (D) 10 N. (E) 7.5 N. (F) 20 N. (G) 5 N. (H) 3.75 N. (I) N/A (J) N/A"
-    target: "Let's think step by step. The force between two point poles is given by \( \frac{m_1 m_2}{\mu_0 4 \pi r^2} \), in analogy to Coulomb’s law. Plugging in the values given in the question, we calculate that the force is approximately 15 N. The answer is (C)."
-  - question: "The coil of a moving coil meter has 100 turns, is 40 mm long and 30 mm wide. The control torque is \( 240 \times 10^{-6} \) N-m on full scale. If magnetic flux density is 1 Wb/m² range of meter is
+    target: "Let's think step by step. The force between two point poles is given by \\( \\frac{m_1 m_2}{\\mu_0 4 \\pi r^2} \\), in analogy to Coulomb’s law. Plugging in the values given in the question, we calculate that the force is approximately 15 N. The answer is (C)."
+  - question: "The coil of a moving coil meter has 100 turns, is 40 mm long and 30 mm wide. The control torque is \\( 240 \\times 10^{-6} \\) N-m on full scale. If magnetic flux density is 1 Wb/m² range of meter is
       (A) 2 mA. (B) 5 mA. (C) 1.5 mA. (D) 0.5 mA. (E) 6 mA. (F) 4 mA. (G) 3 mA. (H) 1 mA. (I) 2.5 mA. (J) 3.5 mA."
-    target: "Let's think step by step. The torque on a coil in a uniform magnetic field is given by \( BANI \), where \( B \) is the magnetic flux density, \( A \) is the area of the coil, \( N \) is the number of turns, and \( I \) is the current. So we have that \( I = \frac{\text{Torque}}{BAN} \), or \( \frac{240 \times 10^{-6}}{1200 \times 10^{-6} \times 100 \times 1} = 2 \times 10^{-3} \). The answer is (A)."
+    target: "Let's think step by step. The torque on a coil in a uniform magnetic field is given by \\( BANI \\), where \\( B \\) is the magnetic flux density, \\( A \\) is the area of the coil, \\( N \\) is the number of turns, and \\( I \\) is the current. So we have that \\( I = \\frac{\\text{Torque}}{BAN} \\), or \\( \\frac{240 \\times 10^{-6}}{1200 \\times 10^{-6} \\times 100 \\times 1} = 2 \\times 10^{-3} \\). The answer is (A)."
   - question: "In an SR latch built from NOR gates, which condition is not allowed
       (A) S=0, R=2 (B) S=2, R=2 (C) S=1, R=1 (D) S=1, R=-1 (E) S=1, R=2 (F) S=0, R=0 (G) S=2, R=0 (H) S=1, R=0 (I) S=2, R=1 (J) S=0, R=1"
-    target: "Let's think step by step. An SR latch is a set-reset latch; in the case where \( S=1 \) and \( R=1 \), the circuit has no stable state; instead a race condition will be produced within the circuit, so the device will be in an undefined state. So \( S=1 \), \( R=1 \) is an illegal input. The answer is (C)."
+    target: "Let's think step by step. An SR latch is a set-reset latch; in the case where \\( S=1 \\) and \\( R=1 \\), the circuit has no stable state; instead a race condition will be produced within the circuit, so the device will be in an undefined state. So \\( S=1 \\), \\( R=1 \\) is an illegal input. The answer is (C)."
   - question: "Two long parallel conductors carry 100 A. If the conductors are separated by 20 mm, the force per meter of length of each conductor will be
       (A) 100 N. (B) 0.01 N. (C) 10 N. (D) 0.5 N. (E) 0.001 N. (F) 5 N. (G) 1 N. (H) 50 N. (I) 0.1 N. (J) 0.0001 N."
-    target: "Let's think step by step. The magnetic force-per-length between two current-carrying conductors is given by \( \frac{\mu_0 I_1 I_2}{2 \pi r} \), where \( r \) is the separation distance and \( I_1 \) and \( I_2 \) are the currents. Plugging in 100 A for \( I_1 \) and \( I_2 \), and 20 mm for \( r \), gives 0.1 N. The answer is (I)."
+    target: "Let's think step by step. The magnetic force-per-length between two current-carrying conductors is given by \\( \\frac{\\mu_0 I_1 I_2}{2 \\pi r} \\), where \\( r \\) is the separation distance and \\( I_1 \\) and \\( I_2 \\) are the currents. Plugging in 100 A for \\( I_1 \\) and \\( I_2 \\), and 20 mm for \\( r \\), gives 0.1 N. The answer is (I)."
   - question: "In a 2 pole lap winding dc machine, the resistance of one conductor is 2Ω and the total number of conductors is 100. Find the total resistance
       (A) 50Ω (B) 1Ω (C) 25Ω (D) 200Ω (E) 10Ω (F) 100Ω (G) 500Ω (H) 150Ω (I) 75Ω (J) 20Ω"
     target: "Let's think step by step. In lap winding, effectively two resistors are connected in parallel, so the actual resistance of each pair is 1 Ohm. Since we have 50 pairs, we get a total resistance of 50 Ohms. The answer is (A)."

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_health.yaml

@@ -3,21 +3,16 @@ description: The following are multiple choice questions (with answers) about he
 fewshot_config:
   sampler: first_n
   samples:
-  - question: "Which of the following is the body cavity that contains the pituitary gland?
-      (A) Ventral (B) Dorsal (C) Buccal (D) Thoracic (E) Pericardial (F) Abdominal (G) Spinal (H) Pelvic (I) Pleural (J) Cranial"
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let’s solve this problem step by step. The pituitary gland is the major endocrine gland attached to the base of the brain, and it is contained in the Cranial cavity. The answer is (J)."
-  - question: "What is the embryological origin of the hyoid bone?
-      (A) The third and fourth pharyngeal arches (B) The fourth pharyngeal arch (C) The third pharyngeal arch (D) The second pharyngeal arch (E) The second, third and fourth pharyngeal arches (F) The first pharyngeal arch (G) The second and third pharyngeal arches (H) The first and third pharyngeal arches (I) The first, second and third pharyngeal arches (J) The first and second pharyngeal arches"
-    target: "Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let’s solve this problem step by step. The hyoid bone, which is also known as the hyooid, is a small U-shaped bone located in the anterior neck. In its resting position, it lies between the base of the mandible and the third cervical vertebrae. We know that the second and the third pharyngeal arches give rise to the horns of the hyoid bone; therefore, the embryological origin of the hyoid bone are the second and the third pharyngeal arches—this information is covered in option (G). Therefore, we conclude that (G) must be the correct answer. The answer is (G)."
-  - question: "What is the difference between a male and a female catheter?
-      (A) Female catheters are used more frequently than male catheters. (B) Male catheters are bigger than female catheters. (C) Male catheters are more flexible than female catheters. (D) Male catheters are made from a different material than female catheters. (E) Female catheters are longer than male catheters. (F) Male catheters are longer than female catheters. (G) Female catheters are bigger than male catheters. (H) Female catheters have a curved shape while male catheters are straight. (I) Male and female catheters are different colours. (J) Male catheters have a smaller diameter than female catheters."
-    target: "Let's think step by step. We refer to Wikipedia articles on clinical knowledge for help. The difference between a male and female catheter is that male catheters tend to be longer than female catheters. The answer is (F)."
-  - question: "How many attempts should you make to cannulate a patient before passing the job on to a senior colleague, according to the medical knowledge of 2020?
-      (A) 1 (B) Unlimited attempts (C) 5 (D) 0, always pass on to a senior colleague (E) 7 (F) 2 (G) 4 (H) 6 (I) 3 (J) 8"
-    target: "Let's think step by step. We refer to Wikipedia articles on clinical knowledge for help. According to the medical protocol as of 2020, you should make two attempts to cannulate a patient before passing the job on to a more-senior practitioner. The answer is (F)."
-  - question: "Why are parvoviruses a highly impactful parasite?
-      (A) They are able to alter the host's DNA (B) Because they have no nucleic acid (C) They can survive in extreme temperatures (D) Only replicate in dividing cells (E) They can infect multiple species (F) They don't require a host to survive (G) Can integrate into host chromosomes (H) N/A (I) N/A (J) N/A"
-    target: "Let's think step by step. We refer to Wikipedia articles on virology for help. Paroviruses are highly impactful because they do not have nucleic acid. The answer is (B)."
+    - question: "Which of the following is the body cavity that contains the pituitary gland? (A) Ventral (B) Dorsal (C) Buccal (D) Thoracic (E) Pericardial (F) Abdominal (G) Spinal (H) Pelvic (I) Pleural (J) Cranial"
+      target: "Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let’s solve this problem step by step. The pituitary gland is the major endocrine gland attached to the base of the brain, and it is contained in the Cranial cavity. The answer is (J)."
+    - question: "What is the embryological origin of the hyoid bone? (A) The third and fourth pharyngeal arches (B) The fourth pharyngeal arch (C) The third pharyngeal arch (D) The second pharyngeal arch (E) The second, third and fourth pharyngeal arches (F) The first pharyngeal arch (G) The second and third pharyngeal arches (H) The first and third pharyngeal arches (I) The first, second and third pharyngeal arches (J) The first and second pharyngeal arches"
+      target: "Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let’s solve this problem step by step. The hyoid bone, which is also known as the hyooid, is a small U-shaped bone located in the anterior neck. In its resting position, it lies between the base of the mandible and the third cervical vertebrae. We know that the second and the third pharyngeal arches give rise to the horns of the hyoid bone; therefore, the embryological origin of the hyoid bone are the second and the third pharyngeal arches—this information is covered in option (G). Therefore, we conclude that (G) must be the correct answer. The answer is (G)."
+    - question: "What is the difference between a male and a female catheter? (A) Female catheters are used more frequently than male catheters. (B) Male catheters are bigger than female catheters. (C) Male catheters are more flexible than female catheters. (D) Male catheters are made from a different material than female catheters. (E) Female catheters are longer than male catheters. (F) Male catheters are longer than female catheters. (G) Female catheters are bigger than male catheters. (H) Female catheters have a curved shape while male catheters are straight. (I) Male and female catheters are different colours. (J) Male catheters have a smaller diameter than female catheters."
+      target: "Let's think step by step. We refer to Wikipedia articles on clinical knowledge for help. The difference between a male and female catheter is that male catheters tend to be longer than female catheters. The answer is (F)."
+    - question: "How many attempts should you make to cannulate a patient before passing the job on to a senior colleague, according to the medical knowledge of 2020? (A) 1 (B) Unlimited attempts (C) 5 (D) 0, always pass on to a senior colleague (E) 7 (F) 2 (G) 4 (H) 6 (I) 3 (J) 8"
+      target: "Let's think step by step. We refer to Wikipedia articles on clinical knowledge for help. According to the medical protocol as of 2020, you should make two attempts to cannulate a patient before passing the job on to a more-senior practitioner. The answer is (F)."
+    - question: "Why are parvoviruses a highly impactful parasite? (A) They are able to alter the host's DNA (B) Because they have no nucleic acid (C) They can survive in extreme temperatures (D) Only replicate in dividing cells (E) They can infect multiple species (F) They don't require a host to survive (G) Can integrate into host chromosomes (H) N/A (I) N/A (J) N/A"
+      target: "Let's think step by step. We refer to Wikipedia articles on virology for help. Paroviruses are highly impactful because they do not have nucleic acid. The answer is (B)."
 group: mmlu_pro_flan_cot_fewshot_other
 include: _mmlu_pro_flan_cot_fewshot_template_yaml
 task: mmlu_pro_flan_cot_fewshot_health

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_other.yaml → lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_miscellaneous.yaml

@@ -20,4 +20,4 @@ fewshot_config:
     target: "Let's think step by step. We refer to Wikipedia for help. Nippon Budokan is an indoor arena in Tokyo, Japan renowned for hosting rock music concerts including Cheap Trick in 1978. 'Cheap Trick at Budokan' became the name of their album. The answer is (J)."
 group: mmlu_pro_flan_cot_fewshot_other
 include: _mmlu_pro_flan_cot_fewshot_template_yaml
-task: mmlu_pro_flan_cot_fewshot_other
+task: mmlu_pro_flan_cot_fewshot_miscellaneous

lm_eval/tasks/mmlu_pro/flan_cot_fewshot/mmlu_pro_physics.yaml

@@ -5,7 +5,7 @@ fewshot_config:
   samples:
   - question: "Say the pupil of your eye has a diameter of 5 mm and you have a telescope with an aperture of 50 cm. How much more light can the telescope gather than your eye?
       (A) 1000 times more (B) 50 times more (C) 5000 times more (D) 500 times more (E) 10000 times more (F) 20000 times more (G) 2000 times more (H) 100 times more (I) 10 times more (J) N/A"
-    target: "Let's think step by step. The amount of light is proportional to the aperture area $A = \pi D^2/4$ for a lens with diameter $D$, so the relative amounts of light between the eye with diameter 5 mm and the telescope with diameter 50 cm is $(50 cm)^2/(5 mm)^2 = 10000$. The answer is (E)."
+    target: "Let's think step by step. The amount of light is proportional to the aperture area $A = \\pi D^2/4$ for a lens with diameter $D$, so the relative amounts of light between the eye with diameter 5 mm and the telescope with diameter 50 cm is $(50 cm)^2/(5 mm)^2 = 10000$. The answer is (E)."
   - question: "Where do most short-period comets come from and how do we know?
       (A) The Kuiper belt; short period comets tend to be in the plane of the solar system just like the Kuiper belt. (B) The asteroid belt; short period comets tend to come from random directions indicating a spherical distribution of comets called the asteroid belt. (C) The asteroid belt; short period comets tend to be in the plane of the solar system just like the asteroid belt. (D) The Oort cloud; short period comets have orbital periods similar to asteroids like Vesta and are found in the plane of the solar system just like the Oort cloud. (E) The Oort Cloud; short period comets tend to come from random directions indicating a spherical distribution of comets called the Oort Cloud. (F) The Oort cloud; short period comets tend to be in the plane of the solar system just like the Oort cloud. (G) The asteroid belt; short period comets have orbital periods similar to asteroids like Vesta and are found in the plane of the solar system just like the asteroid belt. (H) N/A (I) N/A (J) N/A"
     target: "Let's think step by step. Most short-period comets come from the Kuiper belt, and we know because short period comets tend to be in the plane of the solar system, just like the Kuiper belt is. The answer is (A)."

lm_eval/tasks/mmlu_pro/flan_cot_zeroshot/mmlu_pro_other.yaml → lm_eval/tasks/mmlu_pro/flan_cot_zeroshot/mmlu_pro_miscellaneous.yaml

@@ -3,4 +3,4 @@
   \n"
 "group": "mmlu_pro_flan_cot_zeroshot_other"
 "include": "_mmlu_pro_flan_cot_zeroshot_template_yaml"
-"task": "mmlu_pro_flan_cot_zeroshot_other"
+"task": "mmlu_pro_flan_cot_zeroshot_miscellaneous"

lm_eval/tasks/mmlu_pro/generative/mmlu_pro_other.yaml → lm_eval/tasks/mmlu_pro/generative/mmlu_pro_miscellaneous.yaml

@@ -4,5 +4,5 @@
 "group": "mmlu_pro_other_generative"
 "group_alias": "other"
 "include": "_default_template_yaml"
-"task": "mmlu_pro_other_generative"
-"task_alias": "other"
+"task": "mmlu_pro_miscellaneous_generative"
+"task_alias": "miscellaneous"