Skip to content

GitLab

Unverified Commit 815f59e6 authored by Lintang Sutawika's avatar Lintang Sutawika Committed by GitHub
Browse files

Merge pull request #922 from EleutherAI/mmlu_subgroups 

[Refactor] Mmlu subgroups and weight avg
parents 3533e4b9 44124d95
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 804 additions and 1027 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml
View file @ 815f59e6
dataset_name: elementary_mathematics
description: "The following are multiple choice questions (with answers) about elementary\
"dataset_name": "elementary_mathematics"
"description": "The following are multiple choice questions (with answers) about elementary\
\ mathematics.\n\nQ: Olivia used the rule \"Add 11\" to create the number pattern\
\ shown below. 10, 21, 32, 43, 54. Which statement about the number pattern is true?\n\
(A) The 10th number in the pattern will be an even number.\n(B) The number pattern\
......@@ -22,19 +22,20 @@ description: "The following are multiple choice questions (with answers) about e
\ the other choices are incorrect. The answer is (A).\n\nQ: A store sells 107 different\
\ colors of paint. They have 25 cans of each color in storage. The number of cans\
\ of paint the store has in storage can be found using the expression below. 107\
\ \xD7 25. How many cans of paint does the store have in storage?\n(A) 749\n(B)\
\ 2,675\n(C) 2,945\n(D) 4,250\nA: Let's think step by step. We can calculate 107\
\ x 25 = (100 x 25) + (7 x 25) = 2500 + 175 = 2675. The answer is (B).\n\nQ: A total\
\ of 30 players will play basketball at a park. There will be exactly 5 players\
\ on each team. Which statement correctly explains how to find the number of teams\
\ needed?\n(A) Add 5 to 30 to find 35 teams.\n(B) Divide 30 by 5 to find 6 teams.\n\
(C) Multiply 30 and 5 to find 150 teams.\n(D) Subtract 5 from 30 to find 25 teams.\n\
A: Let's think step by step. We want to find the number of teams. We know that there\
\ are 5 players/team, and 30 players. Thus to get the number of teams we divide\
\ players by players/team, so 30 players / 5 players/team = 6 teams. The answer\
\ is (B).\n\nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n\
(B) (5 x 5) + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think\
\ step by step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5\
\ x 4). The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_elementary_mathematics
\ × 25. How many cans of paint does the store have in storage?\n(A) 749\n(B) 2,675\n\
(C) 2,945\n(D) 4,250\nA: Let's think step by step. We can calculate 107 x 25 = (100\
\ x 25) + (7 x 25) = 2500 + 175 = 2675. The answer is (B).\n\nQ: A total of 30 players\
\ will play basketball at a park. There will be exactly 5 players on each team.\
\ Which statement correctly explains how to find the number of teams needed?\n(A)\
\ Add 5 to 30 to find 35 teams.\n(B) Divide 30 by 5 to find 6 teams.\n(C) Multiply\
\ 30 and 5 to find 150 teams.\n(D) Subtract 5 from 30 to find 25 teams.\nA: Let's\
\ think step by step. We want to find the number of teams. We know that there are\
\ 5 players/team, and 30 players. Thus to get the number of teams we divide players\
\ by players/team, so 30 players / 5 players/team = 6 teams. The answer is (B).\n\
\nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n(B) (5 x 5)\
\ + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think step by\
\ step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5 x 4). The\
\ answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_elementary_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml
View file @ 815f59e6
dataset_name: formal_logic
description: "The following are multiple choice questions (with answers) about formal\
"dataset_name": "formal_logic"
"description": "The following are multiple choice questions (with answers) about formal\
\ logic.\n\nQ: Which of the given formulas of PL is the best symbolization of the\
\ following sentence?\nTurtles live long lives and are happy creatures, unless they\
\ are injured.\n(A) (L \u2022 H) \u2261 I (B) (L \u2022 H) \u2228 I (C) L \u2022\
\ (H \u2228 I) (D) L \u2022 (H \u2283 R).\nA: Let's think step by step. We refer\
\ to Wikipedia articles on formal logic for help. Let\u2019s solve this step by\
\ step. Let \u201CL\u201D denote \u201Cliving long\u201D, H \u201Cbeing happy\u201D\
, and \u201CI\u201D \u201Cbeing injured\u201D. Now, consider each choice:\n(A) means\
\ (living long AND being happy) is equivalent to (being injured). \n(B) means (living\
\ long AND being happy) OR (being injured). \n(C) means (living long) AND (being\
\ happy OR being injured). \n(D) means (living long) AND (being happy implies being\
\ R), but what R denotes is not clear.\nObviously, (B) is the best symbolization\
\ of the original sentence. The answer is (B).\n\nQ: Select the best translation\
\ into predicate logic.George borrows Hector's lawnmower. (g: George; h: Hector;\
\ l: Hector's lawnmower; Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh\
\ (D) Bghl\nA: Let's think step by step. We refer to Wikipedia articles on formal\
\ logic for help. Let\u2019s solve this step by step. We are told that \u201CBxyx\u201D\
\ means \u201Cx borrows y from z\u201D. We can rewrite \u201CGeorge borrows Hector's\
\ lawnmower\u201D as \u201CGeorge borrows a lawnmower from Hector\u201D, which can\
\ then be translated into predicate logic as \u201CBglh\u201D. The answer \u201C\
Bglh\u201D appears in (C); therefore, (C) must be the correct answer. The answer\
\ is (C).\n\nQ: \nSelect the best English interpretation of the given arguments\
\ in predicate logic.\nDm\n(\u2200x)(Wx \u2283 ~Dx). \n(\u2200x)Wx \u2228 Ag\t/\
\ (\u2203x)Ax\n(A) Marina is a dancer. Some weaklings are not dancers. Either everything\
\ is a weakling or Georgia plays volleyball. So something plays volleyball. (B)\
\ Marina is a dancer. No weakling is a dancer. Everything is either a weakling or\
\ plays volleyball. So something plays volleyball. (C) Marina is a dancer. Some\
\ weaklings are not dancers. Everything is either a weakling or plays volleyball.\
\ So something plays volleyball. (D) Marina is a dancer. No weakling is a dancer.\
\ Either everything is a weakling or Georgia plays volleyball. So something plays\
\ volleyball.\nA: Let's think step by step. We refer to Wikipedia articles on formal\
\ logic for help. Let\u2019s solve this step by step. Let \u201CD\u201D denote \u201C\
being a dancer\u201D, \u201Cm\u201D denote \u201CMaria\u201D, \u201Cg\u201D denote\
\ \u201CGeorgia\u201D, \u201CW\u201D denote \u201Cweakling\u201D, \u201CA\u201D\
\ denote \u201Cplaying volleyball\u201D. Then, we have the following:\n1. Dm \u2192\
\ Maria is a dance.\n2. (\u2200x)(Wx \u2283 ~Dx). \u2192 For all x, if x is a weakling,\
\ then x is not a dancer. In other words, no weakling is a dancer.\n3. (\u2200x)Wx\
\ \u2228 Ag\t/ (\u2203x)Ax \u2192 For all x, x is a weakling or Georgia plays volleyball.\
\ So there exists an x that plays volleyball. \nOptions (A) and (C) do claim that\
\ some weaklings are not dancers, but the second argument strongly states that no\
\ weakling is a dancer. Thus, we can eliminate them. Option (B) omits the important\
\ detail about Georgia playing volleyball. Option (D) has all the details presented\
\ in the arguments and is the best English interpretation of the arguments. The\
\ answer is (D).\n\nQ: Select the best translation into predicate logic: No people\
\ drive on Mars.\n(A) ~Pd (B) (\u2200x)(Px \u2228 ~Dx) (C) (\u2200x)(Px \u2283 ~Dx)\
\ (D) ~Dp\nA: Let's think step by step. We refer to Wikipedia articles on formal\
\ logic for help. Let\u2019s solve this step by step. Let \u201CP\u201D denote \u201C\
being on Mars\u201D and \u201CD\u201D denote \u201Cdriving on Mars\u201D. Then let\u2019\
s consider each option:\nOption (A): ~Pd \u2192 d is not on Mars.\nOption (B): (\u2200\
x)(Px \u2228 ~Dx) \u2192 For all x, x is on Mars and x do not drive on Mars.\nOption\
\ (C): (\u2200x)(Px \u2283 ~Dx) \u2192 For all x, x is on Mars implies that x do\
\ not drive on Mars.\nOption (D): ~Dp: \u2192 p do not drive on Mars.\nOf all these\
\ options, Option (C) appears to be the best and most meaningful interpretation\
\ of the argument \u201CNo people drive on Mars.\u201D The answer is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_formal_logic
\ are injured.\n(A) (L • H) ≡ I (B) (L • H) ∨ I (C) L • (H ∨ I) (D) L • (H ⊃ R).\n\
A: Let's think step by step. We refer to Wikipedia articles on formal logic for\
\ help. Let’s solve this step by step. Let “L” denote “living long”, H “being happy”,\
\ and “I” “being injured”. Now, consider each choice:\n(A) means (living long AND\
\ being happy) is equivalent to (being injured). \n(B) means (living long AND being\
\ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\
\ injured). \n(D) means (living long) AND (being happy implies being R), but what\
\ R denotes is not clear.\nObviously, (B) is the best symbolization of the original\
\ sentence. The answer is (B).\n\nQ: Select the best translation into predicate\
\ logic.George borrows Hector's lawnmower. (g: George; h: Hector; l: Hector's lawnmower;\
\ Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh (D) Bghl\nA: Let's think\
\ step by step. We refer to Wikipedia articles on formal logic for help. Let’s solve\
\ this step by step. We are told that “Bxyx” means “x borrows y from z”. We can\
\ rewrite “George borrows Hector's lawnmower” as “George borrows a lawnmower from\
\ Hector”, which can then be translated into predicate logic as “Bglh”. The answer\
\ “Bglh” appears in (C); therefore, (C) must be the correct answer. The answer is\
\ (C).\n\nQ: \nSelect the best English interpretation of the given arguments in\
\ predicate logic.\nDm\n(∀x)(Wx ⊃ ~Dx). \n(∀x)Wx ∨ Ag\t/ (∃x)Ax\n(A) Marina is a\
\ dancer. Some weaklings are not dancers. Either everything is a weakling or Georgia\
\ plays volleyball. So something plays volleyball. (B) Marina is a dancer. No weakling\
\ is a dancer. Everything is either a weakling or plays volleyball. So something\
\ plays volleyball. (C) Marina is a dancer. Some weaklings are not dancers. Everything\
\ is either a weakling or plays volleyball. So something plays volleyball. (D) Marina\
\ is a dancer. No weakling is a dancer. Either everything is a weakling or Georgia\
\ plays volleyball. So something plays volleyball.\nA: Let's think step by step.\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ by step. Let “D” denote “being a dancer”, “m” denote “Maria”, “g” denote “Georgia”,\
\ “W” denote “weakling”, “A” denote “playing volleyball”. Then, we have the following:\n\
1. Dm → Maria is a dance.\n2. (∀x)(Wx ⊃ ~Dx). → For all x, if x is a weakling, then\
\ x is not a dancer. In other words, no weakling is a dancer.\n3. (∀x)Wx ∨ Ag\t\
/ (∃x)Ax → For all x, x is a weakling or Georgia plays volleyball. So there exists\
\ an x that plays volleyball. \nOptions (A) and (C) do claim that some weaklings\
\ are not dancers, but the second argument strongly states that no weakling is a\
\ dancer. Thus, we can eliminate them. Option (B) omits the important detail about\
\ Georgia playing volleyball. Option (D) has all the details presented in the arguments\
\ and is the best English interpretation of the arguments. The answer is (D).\n\n\
Q: Select the best translation into predicate logic: No people drive on Mars.\n\
(A) ~Pd (B) (∀x)(Px ∨ ~Dx) (C) (∀x)(Px ⊃ ~Dx) (D) ~Dp\nA: Let's think step by step.\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ by step. Let “P” denote “being on Mars” and “D” denote “driving on Mars”. Then\
\ let’s consider each option:\nOption (A): ~Pd → d is not on Mars.\nOption (B):\
\ (∀x)(Px ∨ ~Dx) → For all x, x is on Mars and x do not drive on Mars.\nOption (C):\
\ (∀x)(Px ⊃ ~Dx) → For all x, x is on Mars implies that x do not drive on Mars.\n\
Option (D): ~Dp: → p do not drive on Mars.\nOf all these options, Option (C) appears\
\ to be the best and most meaningful interpretation of the argument “No people drive\
\ on Mars.” The answer is (C)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_formal_logic"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml
View file @ 815f59e6
dataset_name: global_facts
description: "The following are multiple choice questions (with answers) about global\
\ facts.\n\nQ: As of 2017, how many of the world\u2019s 1-year-old children today\
\ have been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\n\
A: Let's think step by step. We refer to Wikipedia articles on global facts for\
\ help. According to data published by the World Health Organization, the nummber\
\ of 1-year-old children vaccinated in 2017 exceeds 80%. The answer is (A).\n\n\
Q: As of 2019, about what percentage of Americans agree that the state is run for\
\ the benefit of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think\
\ step by step. We refer to Wikipedia articles on global facts for help. In 2019,\
\ about 46% percentage of Americans agree that the state is run for the benefit\
\ of all the people. The answer is (B).\n\nQ: As of 2019, about what percentage\
\ of Russians say it is very important to have free media in our country without\
\ government/state censorship?\n(A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think\
\ step by step. We refer to Wikipedia articles on global facts for help. As of 2019,\
\ about 38% of Russians say it is very important to have free media in our country.\
\ The answer is (A).\n\nQ: As of 2015, since 1990 forests have ____ in Europe and\
\ have ____ in Africa and the Americas.\n(A) increased, increased (B) increased,\
\ decreased (C) decreased, increased (D) decreased, decreased\nA: Let's think step\
\ by step. We refer to Wikipedia articles on global facts for help. As of 2015,\
\ since 1990 forests have increased in Europe and have decreased in Africa and the\
\ Americas. The answer is (B).\n\nQ: Which of the following pairs of statements\
\ are both true (as of 2019)?\n(A) People tend to be optimistic about their own\
\ future and the future of their nation or the world. (B) People tend to be optimistic\
"dataset_name": "global_facts"
"description": "The following are multiple choice questions (with answers) about global\
\ facts.\n\nQ: As of 2017, how many of the world’s 1-year-old children today have\
\ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\nA: Let's\
\ think step by step. We refer to Wikipedia articles on global facts for help. According\
\ to data published by the World Health Organization, the nummber of 1-year-old\
\ children vaccinated in 2017 exceeds 80%. The answer is (A).\n\nQ: As of 2019,\
\ about what percentage of Americans agree that the state is run for the benefit\
\ of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think step by step.\
\ We refer to Wikipedia articles on global facts for help. In 2019, about 46% percentage\
\ of Americans agree that the state is run for the benefit of all the people. The\
\ answer is (B).\n\nQ: As of 2019, about what percentage of Russians say it is very\
\ important to have free media in our country without government/state censorship?\n\
(A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think step by step. We refer to Wikipedia\
\ articles on global facts for help. As of 2019, about 38% of Russians say it is\
\ very important to have free media in our country. The answer is (A).\n\nQ: As\
\ of 2015, since 1990 forests have ____ in Europe and have ____ in Africa and the\
\ Americas.\n(A) increased, increased (B) increased, decreased (C) decreased, increased\
\ (D) decreased, decreased\nA: Let's think step by step. We refer to Wikipedia articles\
\ on global facts for help. As of 2015, since 1990 forests have increased in Europe\
\ and have decreased in Africa and the Americas. The answer is (B).\n\nQ: Which\
\ of the following pairs of statements are both true (as of 2019)?\n(A) People tend\
\ to be optimistic about their own future and the future of their nation or the\
\ world. (B) People tend to be optimistic about their own future but pessimistic\
\ about the future of their nation or the world. (C) People tend to be pessimistic\
\ about their own future but optimistic about the future of their nation or the\
\ world. (D) People tend to be pessimistic about their own future and the future\
\ of their nation or the world.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on global facts for help. As of 2019, most people tend to be optimistic\
\ about their own future but pessimistic about the future of their nation or the\
\ world. (C) People tend to be pessimistic about their own future but optimistic\
\ about the future of their nation or the world. (D) People tend to be pessimistic\
\ about their own future and the future of their nation or the world.\nA: Let's\
\ think step by step. We refer to Wikipedia articles on global facts for help. As\
\ of 2019, most people tend to be optimistic about their own future but pessimistic\
\ about the future of their nation or the world. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_global_facts
\ world. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_global_facts"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml
View file @ 815f59e6
dataset_name: high_school_biology
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_biology"
"description": "The following are multiple choice questions (with answers) about high\
\ school biology.\n\nQ: In animal cells, which of the following represents the most\
\ likely pathway that a secretory protein takes as it is synthesized in a cell?\n\
(A) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory vesicle\u2013\
rough ER (B) Ribosome\u2013Golgi apparatus\u2013rough ER\u2013secretory vesicle\u2013\
plasma membrane (C) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory\
\ vesicle\u2013rough ER (D) Ribosome\u2013rough ER\u2013Golgi apparatus\u2013secretory\
\ vesicle\u2013plasma membrane\nA: Let's think step by step. Protein synthesis starts\
(A) Plasma membrane–Golgi apparatus–ribosome–secretory vesicle–rough ER (B) Ribosome–Golgi\
\ apparatus–rough ER–secretory vesicle–plasma membrane (C) Plasma membrane–Golgi\
\ apparatus–ribosome–secretory vesicle–rough ER (D) Ribosome–rough ER–Golgi apparatus–secretory\
\ vesicle–plasma membrane\nA: Let's think step by step. Protein synthesis starts\
\ at the ribosome, so we can eliminate (A) and (C). The ribosome is often in the\
\ endoplasmic reticulum and moves from there to the Golgi apparatus, where it is\
\ modified and packaged into a vesicle. The vesicle then floats to the plasma membrane\
\ and is secreted. The answer is (D).\n\nQ: A mutation in a bacterial enzyme changed\
\ a previously polar amino acid into a nonpolar amino acid. This amino acid was\
\ located at a site distant from the enzyme\u2019s active site. How might this mutation\
\ alter the enzyme\u2019s substrate specificity?\n(A) By changing the enzyme\u2019\
s pH optimum (B) By changing the enzyme\u2019s location in the cell (C) By changing\
\ the shape of the protein (D) An amino acid change away from the active site cannot\
\ alter the enzyme\u2019s substrate specificity.\nA: Let's think step by step. A\
\ change in an amino acid leads to a change in the primary structure of the protein.\
\ A change in the primary structure may lead to a change in the secondary and the\
\ tertiary structure of the protein. A change in the tertiary structure means a\
\ change in the shape of the protein, so (C) has to be correct. Since the change\
\ does not affect the active site of the enzyme, we do not expect the activity of\
\ the enzyme to be affected. The answer is (C).\n\nQ: Which of the following is\
\ not a way to form recombinant DNA?\n(A) Translation (B) Conjugation (C) Specialized\
\ transduction (D) Transformation\nA: Let's think step by step. The introduction\
\ of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique\
\ in molecular biology and scientific research. There are multiple ways foreign\
\ DNA can be introduced into cells including transformation, transduction, conjugation,\
\ and transfection. In contrast, (A) is not a way to form DNA: during translation\
\ the ribosomes synthesize proteins from RNA. The answer is (A).\n\nQ: Homologous\
\ structures are often cited as evidence for the process of natural selection. All\
\ of the following are examples of homologous structures EXCEPT\n(A) the wings of\
\ a bird and the wings of a bat (B) the flippers of a whale and the arms of a man\
\ (C) the pectoral fins of a porpoise and the flippers of a seal (D) the forelegs\
\ of an insect and the forelimbs of a dog\nA: Let's think step by step. \u200B\u200B\
Homologous structures are similar physical features in organisms that share a common\
\ ancestor \u200B\u200Bbut different functions. Comparisons (B) and (C) are clearly\
\ homologous because they share a common ancestor and the structures serve different\
\ purposes. Bat wings and birg wings are also homologous, while they are both wings,\
\ the forelimbs serve different purposes. Insects and dogs are very far ancestors\
\ since one is vertebrate while the other is invertebrate and the forelimbs serve\
\ the same purpose, so they are not homologous. The answer is (D).\n\nQ: Which of\
\ the following is not known to be involved in the control of cell division?\n(A)\
\ Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast cells\nA: Let's think\
\ step by step. Normal cells move through the cell cycle in a regulated way. At\
\ the checkpoint stage, they use information about their own internal state and\
\ cues from the environment around them to decide whether to proceed with cell division.\
\ Cues like these act by changing the activity of core cell cycle regulators inside\
\ the cell. The most common regulators are cyclins and cyclin-dependent kinases.\
\ Fibroblast cells do not play any role in cell division. The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_biology
\ located at a site distant from the enzyme’s active site. How might this mutation\
\ alter the enzyme’s substrate specificity?\n(A) By changing the enzyme’s pH optimum\
\ (B) By changing the enzyme’s location in the cell (C) By changing the shape of\
\ the protein (D) An amino acid change away from the active site cannot alter the\
\ enzyme’s substrate specificity.\nA: Let's think step by step. A change in an amino\
\ acid leads to a change in the primary structure of the protein. A change in the\
\ primary structure may lead to a change in the secondary and the tertiary structure\
\ of the protein. A change in the tertiary structure means a change in the shape\
\ of the protein, so (C) has to be correct. Since the change does not affect the\
\ active site of the enzyme, we do not expect the activity of the enzyme to be affected.\
\ The answer is (C).\n\nQ: Which of the following is not a way to form recombinant\
\ DNA?\n(A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation\n\
A: Let's think step by step. The introduction of foreign DNA or RNA into bacteria\
\ or eukaryotic cells is a common technique in molecular biology and scientific\
\ research. There are multiple ways foreign DNA can be introduced into cells including\
\ transformation, transduction, conjugation, and transfection. In contrast, (A)\
\ is not a way to form DNA: during translation the ribosomes synthesize proteins\
\ from RNA. The answer is (A).\n\nQ: Homologous structures are often cited as evidence\
\ for the process of natural selection. All of the following are examples of homologous\
\ structures EXCEPT\n(A) the wings of a bird and the wings of a bat (B) the flippers\
\ of a whale and the arms of a man (C) the pectoral fins of a porpoise and the flippers\
\ of a seal (D) the forelegs of an insect and the forelimbs of a dog\nA: Let's think\
\ step by step. ​​Homologous structures are similar physical features in organisms\
\ that share a common ancestor ​​but different functions. Comparisons (B) and (C)\
\ are clearly homologous because they share a common ancestor and the structures\
\ serve different purposes. Bat wings and birg wings are also homologous, while\
\ they are both wings, the forelimbs serve different purposes. Insects and dogs\
\ are very far ancestors since one is vertebrate while the other is invertebrate\
\ and the forelimbs serve the same purpose, so they are not homologous. The answer\
\ is (D).\n\nQ: Which of the following is not known to be involved in the control\
\ of cell division?\n(A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast\
\ cells\nA: Let's think step by step. Normal cells move through the cell cycle in\
\ a regulated way. At the checkpoint stage, they use information about their own\
\ internal state and cues from the environment around them to decide whether to\
\ proceed with cell division. Cues like these act by changing the activity of core\
\ cell cycle regulators inside the cell. The most common regulators are cyclins\
\ and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division.\
\ The answer is (D)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_biology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml
View file @ 815f59e6
dataset_name: high_school_chemistry
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_chemistry"
"description": "The following are multiple choice questions (with answers) about high\
\ school chemistry.\n\nQ: Which of the following is considered an acid anhydride?\n\
(A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3\nA: Let's think step by step. An acid anhydride\
\ is a compound that is derived by removing water from an acid. The chemical formula\
......@@ -45,5 +45,6 @@ description: "The following are multiple choice questions (with answers) about h
\ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\
\ base. Therefore the maximum amount of acid that can be added will be equal to\
\ the amount of acetate ion, or 2 moles. The answer is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_chemistry
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_chemistry"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml
View file @ 815f59e6
dataset_name: high_school_computer_science
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_computer_science"
"description": "The following are multiple choice questions (with answers) about high\
\ school computer science.\n\nQ: Which of the following is an example of the use\
\ of a device on the Internet of Things (IoT) ?\n(A) A car alerts a driver that\
\ it is about to hit an object. (B) A hiker uses a G P S watch to keep track of\
......@@ -26,9 +26,9 @@ description: "The following are multiple choice questions (with answers) about h
\ launched from any web sites visited or files downloaded.\nA: Let's think step\
\ by step. Choice A is incorrect as it only describes network traffic, which an\
\ anonymous browser does not change. Choice B is correct as it correctly describes\
\ how an anonymous browser will prevent saving data on the user\u2019s computer\
\ after the session is ended. Choice C is incorrect because an anonymous browser\
\ will not prevent logging in to email or social media accounts. Choice D is incorrect\
\ how an anonymous browser will prevent saving data on the user’s computer after\
\ the session is ended. Choice C is incorrect because an anonymous browser will\
\ not prevent logging in to email or social media accounts. Choice D is incorrect\
\ because an anonymous browser in itself performs no virus protection. The answer\
\ is (B).\n\nQ: In the program below, the initial value of X is 5 and the initial\
\ value of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\
......@@ -66,5 +66,6 @@ description: "The following are multiple choice questions (with answers) about h
\ its value is greater than 100, regardless of the elements in the list. Choice\
\ D is incorrect because its step 3 does not increment the value of position, so\
\ it will repeat forever. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_computer_science
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_computer_science"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml
View file @ 815f59e6
dataset_name: high_school_european_history
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_european_history"
"description": "The following are multiple choice questions (with answers) about high\
\ school european history.\n\nQ: This question refers to the following information.\n\
Albeit the king's Majesty justly and rightfully is and ought to be the supreme head\
\ of the Church of England, and so is recognized by the clergy of this realm in\
......@@ -34,7 +34,7 @@ description: "The following are multiple choice questions (with answers) about h
\ the corruption in the Church of England. The answer is (D).\n\nQ: This question\
\ refers to the following information.\nRead the following excerpt.\nThe revolutionary\
\ seed had penetrated into every country and spread more or less. It was greatly\
\ developed under the r\xE9gime of the military despotism of Bonaparte. His conquests\
\ developed under the régime of the military despotism of Bonaparte. His conquests\
\ displaced a number of laws, institutions, and customs; broke through bonds sacred\
\ among all nations, strong enough to resist time itself; which is more than can\
\ be said of certain benefits conferred by these innovators.\nThe monarchs will\
......@@ -55,9 +55,9 @@ description: "The following are multiple choice questions (with answers) about h
Let them maintain religious principles in all their purity, and not allow the faith\
\ to be attacked and morality interpreted according to the social contract or the\
\ visions of foolish sectarians.\nLet them suppress Secret Societies; that gangrene\
\ of society.\n\u2014Klemens von Metternich, Political Confession of Faith, 1820\n\
Which of the following was the greatest cause of the fears expressed by Metternich\
\ in the document above?\n(A) The ideas of personal liberty and nationalism conceived\
\ of society.\n—Klemens von Metternich, Political Confession of Faith, 1820\nWhich\
\ of the following was the greatest cause of the fears expressed by Metternich in\
\ the document above?\n(A) The ideas of personal liberty and nationalism conceived\
\ during the Enlightenment resulted in radical revolutions that could spread throughout\
\ Europe. (B) The conquest of Europe by Napoleon led to the creation of new factions\
\ and shifted the European balance of power. (C) The power of monarchs had grown\
......@@ -110,15 +110,15 @@ description: "The following are multiple choice questions (with answers) about h
\ were all turning to the politicians; the famous Nihilists who made Europe tremble-sons\
\ of village priests, of the lower middle class, of tradesmen-could not rise above\
\ the idea of national liberation, and seemed to believe that the world would be\
\ delivered-when they had killed their despot&\u2026\n\"Foolery! They'll never get\
\ out of it with their foolery.\"\nThen, lowering his voice still more, in a few\
\ bitter words he described his old dream of fraternity. He had renounced his rank\
\ and his fortune; he had gone among workmen, only in the hope of seeing at last\
\ the foundation of a new society of labour in common. All the sous in his pockets\
\ had long gone to the urchins of the settlement; he had been as tender as a brother\
\ delivered-when they had killed their despot&…\n\"Foolery! They'll never get out\
\ of it with their foolery.\"\nThen, lowering his voice still more, in a few bitter\
\ words he described his old dream of fraternity. He had renounced his rank and\
\ his fortune; he had gone among workmen, only in the hope of seeing at last the\
\ foundation of a new society of labour in common. All the sous in his pockets had\
\ long gone to the urchins of the settlement; he had been as tender as a brother\
\ with the colliers, smiling at their suspicion, winning them over by his quiet\
\ workmanlike ways and his dislike of chattering. But decidedly the fusion had not\
\ taken place.\nHis voice changed, his eyes grew bright, he fixed them on \xE9tienne,\
\ taken place.\nHis voice changed, his eyes grew bright, he fixed them on étienne,\
\ directly addressing him:\n\"Now, do you understand that? These hatworkers at Marseilles\
\ who have won the great lottery prize of a hundred thousand francs have gone off\
\ at once and invested it, declaring that they are going to live without doing anything!\
......@@ -127,7 +127,7 @@ description: "The following are multiple choice questions (with answers) about h
\ out as much as you like against the rich, you haven't got courage enough to give\
\ back to the poor the money that luck brings you. You will never be worthy of happiness\
\ as long as you own anything, and your hatred of the bourgeois proceeds solely\
\ from an angry desire to be bourgeois yourselves in their place.\"\n\xE9mile Zola,\
\ from an angry desire to be bourgeois yourselves in their place.\"\némile Zola,\
\ French writer, Germinal, 1885\nThe passage displays the direct concern for the\
\ welfare of the working classes that was typically a part of which movement?\n\
(A) Capitalist (B) Scientific (C) Communist (D) Existentialist\nA: Let's think step\
......@@ -156,13 +156,14 @@ description: "The following are multiple choice questions (with answers) about h
\ whether Jewish, Christian or Turkish, appear to me no other than human inventions,\
\ set up to terrify and enslave mankind, and monopolize power and profit.\nI do\
\ not mean by this declaration to condemn those who believe otherwise; they have\
\ the same right to their belief as I have to mine.\n\u2014Thomas Paine, The Age\
\ of Reason, 1794\u20131795\nWhich of the following Enlightenment philosophes designed\
\ a system of checks and balances for government to avoid abuses of power?\n(A)\
\ Jean Jacques Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\n\
A: Let's think step by step. We refer to Wikipedia articles on european history\
\ for help. Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\
\ the same right to their belief as I have to mine.\n—Thomas Paine, The Age of Reason,\
\ 1794–1795\nWhich of the following Enlightenment philosophes designed a system\
\ of checks and balances for government to avoid abuses of power?\n(A) Jean Jacques\
\ Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\nA: Let's\
\ think step by step. We refer to Wikipedia articles on european history for help.\
\ Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\
\ against the monoplization of power and advocated for a system of checks and balances\
\ in government to prevent the rise of despotism. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_european_history
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_european_history"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml
View file @ 815f59e6
dataset_name: high_school_geography
description: 'The following are multiple choice questions (with answers) about high
school geography.
Q: Which one of the following items is an example of nonmaterial culture?
(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not
physical objects. The answer is (C).
Q: During the third stage of the demographic transition model, which of the following
is true?
(A) Birth rates increase and population growth rate is less rapid. (B) Birth rates
decline and population growth rate is less rapid. (C) Birth rates increase and population
growth rate increases. (D) Birth rates decrease and population growth rate increases.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
The demographic transition model models the five different stages of population
growth as a country goes through economic development, where the third stage refers
to a period of declining birth rates and lower population growth. The answer is
(B).
Q: The practice of hiring a foreign third-party service provider to run an operation
is called
(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
"Offshoring" literally means to move or base some of the activities or processes
of a company to a foreign country. The answer is (B).
Q: Which of the following statements is NOT accurate regarding the services provided
by local governments in the United States?
(A) Duplication of efforts occurs often. (B) Social problems of the central city
spill over into the surrounding residential suburbs. (C) Inefficiency in providing
services occurs often. (D) One neighborhood''s efforts to reduce pollution are always
supported by neighboring communities.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
There may be economic, social or political reasons for two neighboring communities
and their local governments not agreeing to pollution reduction efforts initiated
by one of them. The answer is (D).
Q: The rate of natural increase of a population is found by subtracting the
(A) crude death rate from the crude birth date. (B) crude birth rate from the crude
death rate. (C) doubling time from the crude birth rate. (D) fertility rate from
the crude death rate.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
The difference between number of births and deaths gives the population increase
at any given time. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_geography
"dataset_name": "high_school_geography"
"description": "The following are multiple choice questions (with answers) about high\
\ school geography.\n\nQ: Which one of the following items is an example of nonmaterial\
\ culture?\n(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).\n\
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
\ Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not\
\ physical objects. The answer is (C).\n\nQ: During the third stage of the demographic\
\ transition model, which of the following is true?\n(A) Birth rates increase and\
\ population growth rate is less rapid. (B) Birth rates decline and population growth\
\ rate is less rapid. (C) Birth rates increase and population growth rate increases.\
\ (D) Birth rates decrease and population growth rate increases.\nA: Let's think\
\ step by step. We refer to Wikipedia articles on geography for help. The demographic\
\ transition model models the five different stages of population growth as a country\
\ goes through economic development, where the third stage refers to a period of\
\ declining birth rates and lower population growth. The answer is (B).\n\nQ: The\
\ practice of hiring a foreign third-party service provider to run an operation\
\ is called\n(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.\n\
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
\ \"Offshoring\" literally means to move or base some of the activities or processes\
\ of a company to a foreign country. The answer is (B).\n\nQ: Which of the following\
\ statements is NOT accurate regarding the services provided by local governments\
\ in the United States?\n(A) Duplication of efforts occurs often. (B) Social problems\
\ of the central city spill over into the surrounding residential suburbs. (C) Inefficiency\
\ in providing services occurs often. (D) One neighborhood's efforts to reduce pollution\
\ are always supported by neighboring communities.\nA: Let's think step by step.\
\ We refer to Wikipedia articles on geography for help. There may be economic, social\
\ or political reasons for two neighboring communities and their local governments\
\ not agreeing to pollution reduction efforts initiated by one of them. The answer\
\ is (D).\n\nQ: The rate of natural increase of a population is found by subtracting\
\ the\n(A) crude death rate from the crude birth date. (B) crude birth rate from\
\ the crude death rate. (C) doubling time from the crude birth rate. (D) fertility\
\ rate from the crude death rate.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on geography for help. The difference between number of births and deaths\
\ gives the population increase at any given time. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_geography"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml
View file @ 815f59e6
dataset_name: high_school_government_and_politics
description: 'The following are multiple choice questions (with answers) about high
school government and politics.
Q: Which of the following best states an argument made by James Madison in The Federalist
number 10?
(A) Honest politicians can prevent factions from developing. (B) Factions are more
likely to occur in large republics than in small ones. (C) The negative effects
of factionalism can be reduced by a republican government. (D) Free elections are
the people''s best defense against factionalism.
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. In the Federalist number 10, James Madison advocated for a representative
republican form of government to guard against factionalism. The answer is (C).
Q: The term "budget deficit" refers to the
(A) annual increase in federal spending on the military (B) amount of interest on
the national debt (C) difference between the initial budget proposals made by the
president and Congress (D) amount the government spends in excess of its revenues
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. When the goverment spends more than it earns, their difference is the
budget deficit. The answer is (D).
Q: Which of the following statements about cabinet departments is FALSE?
(A) They are established by the legislative branch. (B) Their members often don''t
have much influence over presidential decisions. (C) They cannot all be run by leaders
who belong to the same political party the president does. (D) Not every federal
agency is a cabinet department.
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. There is no law stipulating that some cabinet department leaders have
to belong to a political party different from that of the president. The answer
is (C).
Q: Which of the following cases established the precedent that a defendant must
be informed of the right to remain silent, the right to a lawyer, and protection
from self-incrimination?
(A) Weeks v. United States (B) Betts v. Brady (C) Mapp v. Ohio (D) Miranda v. Arizona
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. In the landmark Miranda v. Arizona in 1966, the US Supreme Court, based
on the Fifth and Sixth Amendment of the US Constitution, guaranteed a defendant''s
right to an attorney and protection from self-incrimination. The answer is (D).
Q: Uncertainty over the limits to presidential power is caused primarily by the
fact that
(A) the constitutional definition of those powers is broad and unspecific (B) most
people agree that the Constitution places too many limits on presidential power
(C) the Supreme Court consistently refuses to rule on cases concerning presidential
powers (D) constitutional amendments have greatly increased presidential powers
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. The US Constitution is not very specific about the powers of the president,
leading to uncertainty over its limits. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_government_and_politics
"dataset_name": "high_school_government_and_politics"
"description": "The following are multiple choice questions (with answers) about high\
\ school government and politics.\n\nQ: Which of the following best states an argument\
\ made by James Madison in The Federalist number 10?\n(A) Honest politicians can\
\ prevent factions from developing. (B) Factions are more likely to occur in large\
\ republics than in small ones. (C) The negative effects of factionalism can be\
\ reduced by a republican government. (D) Free elections are the people's best defense\
\ against factionalism.\nA: Let's think step by step. We refer to Wikipedia articles\
\ on government and politics for help. In the Federalist number 10, James Madison\
\ advocated for a representative republican form of government to guard against\
\ factionalism. The answer is (C).\n\nQ: The term \"budget deficit\" refers to the\n\
(A) annual increase in federal spending on the military (B) amount of interest on\
\ the national debt (C) difference between the initial budget proposals made by\
\ the president and Congress (D) amount the government spends in excess of its revenues\n\
A: Let's think step by step. We refer to Wikipedia articles on government and politics\
\ for help. When the goverment spends more than it earns, their difference is the\
\ budget deficit. The answer is (D).\n\nQ: Which of the following statements about\
\ cabinet departments is FALSE?\n(A) They are established by the legislative branch.\
\ (B) Their members often don't have much influence over presidential decisions.\
\ (C) They cannot all be run by leaders who belong to the same political party the\
\ president does. (D) Not every federal agency is a cabinet department.\nA: Let's\
\ think step by step. We refer to Wikipedia articles on government and politics\
\ for help. There is no law stipulating that some cabinet department leaders have\
\ to belong to a political party different from that of the president. The answer\
\ is (C).\n\nQ: Which of the following cases established the precedent that a defendant\
\ must be informed of the right to remain silent, the right to a lawyer, and protection\
\ from self-incrimination?\n(A) Weeks v. United States (B) Betts v. Brady (C) Mapp\
\ v. Ohio (D) Miranda v. Arizona\nA: Let's think step by step. We refer to Wikipedia\
\ articles on government and politics for help. In the landmark Miranda v. Arizona\
\ in 1966, the US Supreme Court, based on the Fifth and Sixth Amendment of the US\
\ Constitution, guaranteed a defendant's right to an attorney and protection from\
\ self-incrimination. The answer is (D).\n\nQ: Uncertainty over the limits to presidential\
\ power is caused primarily by the fact that\n(A) the constitutional definition\
\ of those powers is broad and unspecific (B) most people agree that the Constitution\
\ places too many limits on presidential power (C) the Supreme Court consistently\
\ refuses to rule on cases concerning presidential powers (D) constitutional amendments\
\ have greatly increased presidential powers\nA: Let's think step by step. We refer\
\ to Wikipedia articles on government and politics for help. The US Constitution\
\ is not very specific about the powers of the president, leading to uncertainty\
\ over its limits. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_government_and_politics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml
View file @ 815f59e6
dataset_name: high_school_macroeconomics
description: 'The following are multiple choice questions (with answers) about high
school macroeconomics.
Q: Which of the following policies best describes supply-side fiscal policy?
(A) An increase in the money supply (B) Increased government spending (C) Lower
taxes on research and development of new technology (D) Higher taxes on household
income
A: Let''s think step by step. We refer to Wikipedia articles on macroeconomics for
help. Supply-side fiscal policy stimulates the economy by encouraging more production
of goods and services through reduction in taxes and deregulation. The answer is
(C).
Q: The short-run Phillips curve indicates a
(A) direct relation between unemployment and inflation (B) direct relation between
price and quantity demanded (C) inverse relation between price and quantity demanded
(D) inverse relation between unemployment and inflation
A: Let''s think step by step. We refer to Wikipedia articles on macroeconomics for
help. The short-run Phillips curve shows that whenever unemployment decreases below
a natural level, the inflation starts increasing, and vice-versa. The answer is
(D).
Q: Holding all else equal which of the following monetary policies would be used
to boost U.S. exports?
(A) Increasing the discount rate (B) Increasing the reserve ratio (C) Buying government
securities (D) Lowering tariffs
A: Let''s think step by step. We refer to Wikipedia articles on macroeconomics for
help. Buying government securities leads to reduction in demand for US dollars from
foreign buyers, thereby making it cheaper and hence making US exports more attractive.
The answer is (C).
Q: A federal deficit occurs when
(A) exports exceed imports. (B) imports exceed exports. (C) federal tax collections
exceed spending. (D) federal spending exceeds federal tax revenues.
A: Let''s think step by step. We refer to Wikipedia articles on macroeconomics for
help. A federal deficit occurs when federal spending exceeds federal income which
is primarily from tax revenues. The answer is (D).
Q: Which of the following is not included in the U.S. GDP?
(A) The U.S. military opens a new base in a foreign country with 1000 U.S. personnel.
(B) Japanese consumers buy thousands of CDs produced in the United States. (C) An
American pop singer performs a sold-out concert in Paris. (D) A French theatrical
production tours dozens of American cities.
A: Let''s think step by step. We refer to Wikipedia articles on macroeconomics for
help. The economic transactions related to the performance of the American pop-singer
in Paris happens entirely outside the U.S. and hence is not included in the GDP
numbers. The answer is (C).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_macroeconomics
"dataset_name": "high_school_macroeconomics"
"description": "The following are multiple choice questions (with answers) about high\
\ school macroeconomics.\n\nQ: Which of the following policies best describes supply-side\
\ fiscal policy?\n(A) An increase in the money supply (B) Increased government spending\
\ (C) Lower taxes on research and development of new technology (D) Higher taxes\
\ on household income\nA: Let's think step by step. We refer to Wikipedia articles\
\ on macroeconomics for help. Supply-side fiscal policy stimulates the economy by\
\ encouraging more production of goods and services through reduction in taxes and\
\ deregulation. The answer is (C).\n\nQ: The short-run Phillips curve indicates\
\ a\n(A) direct relation between unemployment and inflation (B) direct relation\
\ between price and quantity demanded (C) inverse relation between price and quantity\
\ demanded (D) inverse relation between unemployment and inflation\nA: Let's think\
\ step by step. We refer to Wikipedia articles on macroeconomics for help. The short-run\
\ Phillips curve shows that whenever unemployment decreases below a natural level,\
\ the inflation starts increasing, and vice-versa. The answer is (D).\n\nQ: Holding\
\ all else equal which of the following monetary policies would be used to boost\
\ U.S. exports?\n(A) Increasing the discount rate (B) Increasing the reserve ratio\
\ (C) Buying government securities (D) Lowering tariffs\nA: Let's think step by\
\ step. We refer to Wikipedia articles on macroeconomics for help. Buying government\
\ securities leads to reduction in demand for US dollars from foreign buyers, thereby\
\ making it cheaper and hence making US exports more attractive. The answer is (C).\n\
\nQ: A federal deficit occurs when\n(A) exports exceed imports. (B) imports exceed\
\ exports. (C) federal tax collections exceed spending. (D) federal spending exceeds\
\ federal tax revenues.\nA: Let's think step by step. We refer to Wikipedia articles\
\ on macroeconomics for help. A federal deficit occurs when federal spending exceeds\
\ federal income which is primarily from tax revenues. The answer is (D).\n\nQ:\
\ Which of the following is not included in the U.S. GDP?\n(A) The U.S. military\
\ opens a new base in a foreign country with 1000 U.S. personnel. (B) Japanese consumers\
\ buy thousands of CDs produced in the United States. (C) An American pop singer\
\ performs a sold-out concert in Paris. (D) A French theatrical production tours\
\ dozens of American cities.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on macroeconomics for help. The economic transactions related to the\
\ performance of the American pop-singer in Paris happens entirely outside the U.S.\
\ and hence is not included in the GDP numbers. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_macroeconomics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml
View file @ 815f59e6
dataset_name: high_school_mathematics
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_mathematics"
"description": "The following are multiple choice questions (with answers) about high\
\ school mathematics.\n\nQ: Simplify and write the result with a rational denominator:\
\ $$\\sqrt{\\sqrt[3]{\\sqrt{\\frac{1}{729}}}}$$\n(A) \\frac{3\\sqrt{3}}{3} (B) \\\
frac{1}{3} (C) \\sqrt{3} (D) \\frac{\\sqrt{3}}{3}\nA: Let's think step by step.\
......@@ -13,7 +13,7 @@ description: "The following are multiple choice questions (with answers) about h
\ of $9600/300=32=2^5$. Since at this interest rate it takes six years for it to\
\ double, it will take $5*6=30$ years to grow to $\\$9600$. The answer is (C).\n\
\nQ: Ten students take a biology test and receive the following scores: 45, 55,\
\ 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students\u2019 test scores?\n\
\ 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students’ test scores?\n\
(A) 55 (B) 60 (C) 62 (D) 65\nA: Let's think step by step. There are 10 students\
\ and the sum of their scores is $45 + 55 + 50 + 70 + 65 + 80 + 40 + 90 + 70 + 85\
\ = 650$, the mean is $650/10=65$. The answer is (D).\n\nQ: The variable $x$ varies\
......@@ -32,5 +32,6 @@ description: "The following are multiple choice questions (with answers) about h
\ dance.)\n(A) 3 (B) 15 (C) 6 (D) 5\nA: Let's think step by step. The least common\
\ multiple of 2, 3 and 5 is 30, so during a 7 minute dance, all the three lights\
\ will come on at the same time $2*7+1=15$ times. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_mathematics
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml
View file @ 815f59e6
dataset_name: high_school_microeconomics
description: 'The following are multiple choice questions (with answers) about high
school microeconomics.
Q: Which of the following is necessarily a characteristic of oligopoly?
(A) Free entry into and exit from the market (B) A few large producers (C) One producer
of a good with no close substitutes (D) A homogenous product
A: Let''s think step by step. We refer to Wikipedia articles on microeconomics for
help. An oligopoly is when a market is dominated by just one or a few number of
sellers or producers. To get oligopoly, the market should have high barriers to
new entry, and the product has differentiation. The answer is (B).
Q: If the government subsidizes producers in a perfectly competitive market, then
(A) the demand for the product will increase (B) the demand for the product will
decrease (C) the consumer surplus will increase (D) the consumer surplus will decrease
A: Let''s think step by step. We refer to Wikipedia articles on microeconomics for
help. (A) and (B) are wrong because the demand curve does not change at all. If
the government subsidizes producers, the supply will increase, and thus the consumer
surplus also increases. The answer is (C).
Q: Which of the following is true of a price floor?
(A) The price floor shifts the demand curve to the left. (B) An effective floor
creates a shortage of the good. (C) The price floor shifts the supply curve of the
good to the right. (D) To be an effective floor, it must be set above the equilibrium
price.
A: Let''s think step by step. We refer to Wikipedia articles on microeconomics for
help. Price floor does not shift the demand or shift curve. An effective price floor
should be set above the equilibrium price, otherwise the market bears and the floor
does not have effective effect. The answer is (D).
Q: The concentration ratio for a monopoly is
(A) 0 (B) 5 (C) 10 (D) 100
A: Let''s think step by step. We refer to Wikipedia articles on microeconomics for
help. The concentration ratio is calculated as the sum of market share of a specific
number of largest companies. Monopoly means one company or entity controls the entire
market, therefore, the concentration ratio is 100 percent. The answer is (D).
Q: In a competitive labor market for housepainters, which of the following would
increase the demand for housepainters?
(A) An effective minimum wage imposed on this labor market. (B) An increase in the
price of gallons of paint. (C) An increase in the construction of new houses. (D)
An increase in the price of mechanical painters so long as the output effect exceeds
the substitution effect.
A: Let''s think step by step. We refer to Wikipedia articles on microeconomics for
help. An increase in the construction of new houses means an increase demand of
in-house painting, thus increases the demand for housepainters. The answer is (C).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_microeconomics
"dataset_name": "high_school_microeconomics"
"description": "The following are multiple choice questions (with answers) about high\
\ school microeconomics.\n\nQ: Which of the following is necessarily a characteristic\
\ of oligopoly?\n(A) Free entry into and exit from the market (B) A few large producers\
\ (C) One producer of a good with no close substitutes (D) A homogenous product\n\
A: Let's think step by step. We refer to Wikipedia articles on microeconomics for\
\ help. An oligopoly is when a market is dominated by just one or a few number of\
\ sellers or producers. To get oligopoly, the market should have high barriers to\
\ new entry, and the product has differentiation. The answer is (B).\n\nQ: If the\
\ government subsidizes producers in a perfectly competitive market, then\n(A) the\
\ demand for the product will increase (B) the demand for the product will decrease\
\ (C) the consumer surplus will increase (D) the consumer surplus will decrease\n\
A: Let's think step by step. We refer to Wikipedia articles on microeconomics for\
\ help. (A) and (B) are wrong because the demand curve does not change at all. If\
\ the government subsidizes producers, the supply will increase, and thus the consumer\
\ surplus also increases. The answer is (C).\n\nQ: Which of the following is true\
\ of a price floor?\n(A) The price floor shifts the demand curve to the left. (B)\
\ An effective floor creates a shortage of the good. (C) The price floor shifts\
\ the supply curve of the good to the right. (D) To be an effective floor, it must\
\ be set above the equilibrium price.\nA: Let's think step by step. We refer to\
\ Wikipedia articles on microeconomics for help. Price floor does not shift the\
\ demand or shift curve. An effective price floor should be set above the equilibrium\
\ price, otherwise the market bears and the floor does not have effective effect.\
\ The answer is (D).\n\nQ: The concentration ratio for a monopoly is\n(A) 0 (B)\
\ 5 (C) 10 (D) 100\nA: Let's think step by step. We refer to Wikipedia articles\
\ on microeconomics for help. The concentration ratio is calculated as the sum of\
\ market share of a specific number of largest companies. Monopoly means one company\
\ or entity controls the entire market, therefore, the concentration ratio is 100\
\ percent. The answer is (D).\n\nQ: In a competitive labor market for housepainters,\
\ which of the following would increase the demand for housepainters?\n(A) An effective\
\ minimum wage imposed on this labor market. (B) An increase in the price of gallons\
\ of paint. (C) An increase in the construction of new houses. (D) An increase in\
\ the price of mechanical painters so long as the output effect exceeds the substitution\
\ effect.\nA: Let's think step by step. We refer to Wikipedia articles on microeconomics\
\ for help. An increase in the construction of new houses means an increase demand\
\ of in-house painting, thus increases the demand for housepainters. The answer\
\ is (C)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_microeconomics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml
View file @ 815f59e6
dataset_name: high_school_physics
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_physics"
"description": "The following are multiple choice questions (with answers) about high\
\ school physics.\n\nQ: A microwave oven is connected to an outlet, 120 V, and draws\
\ a current of 2 amps. At what rate is energy being used by the microwave oven?\n\
(A) 10 W (B) 30 W (C) 60 W (D) 240 W\nA: Let's think step by step. Rate of energy\
\ usage is known as power; in an dissipative electrical circuit, power is given\
\ by voltage times current. So in our case, the power is 120 V times 2 amps, or\
\ 240 W. The answer is (D).\n\nQ: A point charge, Q = +1 mC, is fixed at the origin.\
\ How much work is required to move a charge, Q = +8 \xB5C, from the point (0, 4\
\ meters) to the point (3 meters, 0)?\n(A) 3.5 J (B) 6.0 J (C) 22.5 J (D) 40 J\n\
A: Let's think step by step. To calculate the work required to move a charge from\
\ one location to another in a fixed electric field, it is enough to calculate the\
\ potential difference between the two locations. Here, the potential only depends\
\ on the distance between the charges; it\u2019s $k q_1 q_2 / r$, where $k$ is Coulomb\u2019\
s constant. Plugging in values $q_1 = $ 1 mC, $q_2 = 8 \\mu$ C, gives the answer\
\ as 5.992 J, which rounds to 6 J. The answer is (B).\n\nQ: Which of the following\
\ conditions will ensure that angular momentum is conserved? I. Conservation of\
\ linear momentum II. Zero net external force III. Zero net external torque\n(A)\
\ I and II only (B) I and III only (C) II and III only (D) III only\nA: Let's think\
\ step by step. Torque is defined as the change in angular momentum; if there is\
\ zero external torque, angular momentum is conserved. The answer is (D).\n\nQ:\
\ A photocell of work function \u03D5 = 2eV is connected to a resistor in series.\
\ Light of frequency f = 1 \xD7 10^15 Hz hits a metal plate of the photocell. If\
\ the power of the light is P = 100 W, what is the current through the resistor?\n\
(A) 2:00 AM (B) 6:00 AM (C) 12:00 AM (D) 24 A\nA: Let's think step by step. The\
\ only answer above which has units of current is D, 24 A. The answer is (D).\n\n\
Q: A pipe full of air is closed at one end. A standing wave is produced in the pipe,\
\ causing the pipe to sound a note. Which of the following is a correct statement\
\ about the wave\u2019s properties at the closed end of the pipe?\n(A) The pressure\
\ is at a node, but the particle displacement is at an antinode. (B) The pressure\
\ is at an antinode, but the particle displacement is at a node. (C) The pressure\
\ and the particle displacement are both at nodes. (D) The pressure and the particle\
\ displacement are both at antinodes.\nA: Let's think step by step. At the closed\
\ end of the pipe, the particles cannot have any net displacement because the pipe\
\ closure stops them. So the particle displacement is at a node. This closure also\
\ causes the pressure to be maximal, i.e. an antinode. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_physics
\ How much work is required to move a charge, Q = +8 µC, from the point (0, 4 meters)\
\ to the point (3 meters, 0)?\n(A) 3.5 J (B) 6.0 J (C) 22.5 J (D) 40 J\nA: Let's\
\ think step by step. To calculate the work required to move a charge from one location\
\ to another in a fixed electric field, it is enough to calculate the potential\
\ difference between the two locations. Here, the potential only depends on the\
\ distance between the charges; it’s $k q_1 q_2 / r$, where $k$ is Coulomb’s constant.\
\ Plugging in values $q_1 = $ 1 mC, $q_2 = 8 \\mu$ C, gives the answer as 5.992\
\ J, which rounds to 6 J. The answer is (B).\n\nQ: Which of the following conditions\
\ will ensure that angular momentum is conserved? I. Conservation of linear momentum\
\ II. Zero net external force III. Zero net external torque\n(A) I and II only (B)\
\ I and III only (C) II and III only (D) III only\nA: Let's think step by step.\
\ Torque is defined as the change in angular momentum; if there is zero external\
\ torque, angular momentum is conserved. The answer is (D).\n\nQ: A photocell of\
\ work function ϕ = 2eV is connected to a resistor in series. Light of frequency\
\ f = 1 × 10^15 Hz hits a metal plate of the photocell. If the power of the light\
\ is P = 100 W, what is the current through the resistor?\n(A) 2:00 AM (B) 6:00\
\ AM (C) 12:00 AM (D) 24 A\nA: Let's think step by step. The only answer above which\
\ has units of current is D, 24 A. The answer is (D).\n\nQ: A pipe full of air is\
\ closed at one end. A standing wave is produced in the pipe, causing the pipe to\
\ sound a note. Which of the following is a correct statement about the wave’s properties\
\ at the closed end of the pipe?\n(A) The pressure is at a node, but the particle\
\ displacement is at an antinode. (B) The pressure is at an antinode, but the particle\
\ displacement is at a node. (C) The pressure and the particle displacement are\
\ both at nodes. (D) The pressure and the particle displacement are both at antinodes.\n\
A: Let's think step by step. At the closed end of the pipe, the particles cannot\
\ have any net displacement because the pipe closure stops them. So the particle\
\ displacement is at a node. This closure also causes the pressure to be maximal,\
\ i.e. an antinode. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_physics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml
View file @ 815f59e6
dataset_name: high_school_psychology
description: 'The following are multiple choice questions (with answers) about high
school psychology.
Q: Pascale is interested in the processing strategies children use to learn new
information. Pascale would best be classified as what type of psychologist?
(A) sociocultural (B) clinical (C) cognitive (D) behaviorist
A: Let''s think step by step. We refer to Wikipedia articles on psychology for help.
Sociocultural psychologist focuses on the effect of societal factors on people.
Clinical psychologist focuses on people with mental issues. Cognitive psychologist
focuses on how people think and learn, including the processing strategies. Behaviorist
focuses more on the environment and experience effect on people. The answer is (C).
Q: According to Caplan''s model of consultee-centered case consultation, the consultant
is primarily interested in
(A) identifying the causes and solutions of the client''s presenting problems (B)
identifying and eliminating the causes of the consultee''s difficulties in handling
a problem (C) establishing a hierarchy of authority to enable effective decision
making (D) presenting a single, well-defined and unambiguous course of action for
the consultant to overcome skills deficits
A: Let''s think step by step. We refer to Wikipedia articles on psychology for help.
Caplan defines two type of consultation. Client-centered case consultation aims
to handle client''s problems, while consultee-centered case consultation aims to
identify the reason of client''s difficulty to solve problems. The answer is (B).
Q: According to the Individuals with Disabilities Education Improvement Act, which
of the following must an educational agency do before it changes the educational
placement of a student with a disability?
(A) Give the child a trial period in the new environment (B) Notify the parents
in writing (C) Obtain school board approval (D) Obtain parental consent
A: Let''s think step by step. We refer to Wikipedia articles on psychology for help.
When the decision to change the educational placement of a student with a disability
is made, the educational agency must notify the parents in writing on that date.
The answer is (B).
Q: While swimming in the ocean, Ivan is frightened by a dark shadow in the water
even before he has the chance to identify what the shadow is. The synaptic connections
taking place during this incident of fright are best described by which of the following?
(A) Messages are sent from the thalamus directly to the amygdala. (B) Messages are
sent from the thalamus to the "what" and "where" pathways. (C) Messages are sent
from the parasympathetic nervous system to the cerebral cortex. (D) Messages are
sent from the frontal lobes to the pituitary gland.
A: Let''s think step by step. We refer to Wikipedia articles on psychology for help.
Our neural system has a mechanism that can respond immediate emotional signal before
going to the thought center. In the Ivan''s case, messages travel directly from
thalamus to amygdala. The answer is (A).
Q: Ani believes that her attitudes and behavior play a central role in what happens
to her. Such a belief is likely to be associated with
(A) a strong superego. (B) low self-esteem. (C) low self-efficacy. (D) an internal
locus of control.
A: Let''s think step by step. We refer to Wikipedia articles on psychology for help.
People with an external locus of control believes fate and luck play an important
role in their lives, while people with an internal locus of control believes they
control their lives. The answer is (D).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_psychology
"dataset_name": "high_school_psychology"
"description": "The following are multiple choice questions (with answers) about high\
\ school psychology.\n\nQ: Pascale is interested in the processing strategies children\
\ use to learn new information. Pascale would best be classified as what type of\
\ psychologist?\n(A) sociocultural (B) clinical (C) cognitive (D) behaviorist\n\
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
\ Sociocultural psychologist focuses on the effect of societal factors on people.\
\ Clinical psychologist focuses on people with mental issues. Cognitive psychologist\
\ focuses on how people think and learn, including the processing strategies. Behaviorist\
\ focuses more on the environment and experience effect on people. The answer is\
\ (C).\n\nQ: According to Caplan's model of consultee-centered case consultation,\
\ the consultant is primarily interested in\n(A) identifying the causes and solutions\
\ of the client's presenting problems (B) identifying and eliminating the causes\
\ of the consultee's difficulties in handling a problem (C) establishing a hierarchy\
\ of authority to enable effective decision making (D) presenting a single, well-defined\
\ and unambiguous course of action for the consultant to overcome skills deficits\n\
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
\ Caplan defines two type of consultation. Client-centered case consultation aims\
\ to handle client's problems, while consultee-centered case consultation aims to\
\ identify the reason of client's difficulty to solve problems. The answer is (B).\n\
\nQ: According to the Individuals with Disabilities Education Improvement Act, which\
\ of the following must an educational agency do before it changes the educational\
\ placement of a student with a disability?\n(A) Give the child a trial period in\
\ the new environment (B) Notify the parents in writing (C) Obtain school board\
\ approval (D) Obtain parental consent\nA: Let's think step by step. We refer to\
\ Wikipedia articles on psychology for help. When the decision to change the educational\
\ placement of a student with a disability is made, the educational agency must\
\ notify the parents in writing on that date. The answer is (B).\n\nQ: While swimming\
\ in the ocean, Ivan is frightened by a dark shadow in the water even before he\
\ has the chance to identify what the shadow is. The synaptic connections taking\
\ place during this incident of fright are best described by which of the following?\n\
(A) Messages are sent from the thalamus directly to the amygdala. (B) Messages are\
\ sent from the thalamus to the \"what\" and \"where\" pathways. (C) Messages are\
\ sent from the parasympathetic nervous system to the cerebral cortex. (D) Messages\
\ are sent from the frontal lobes to the pituitary gland.\nA: Let's think step by\
\ step. We refer to Wikipedia articles on psychology for help. Our neural system\
\ has a mechanism that can respond immediate emotional signal before going to the\
\ thought center. In the Ivan's case, messages travel directly from thalamus to\
\ amygdala. The answer is (A).\n\nQ: Ani believes that her attitudes and behavior\
\ play a central role in what happens to her. Such a belief is likely to be associated\
\ with\n(A) a strong superego. (B) low self-esteem. (C) low self-efficacy. (D) an\
\ internal locus of control.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on psychology for help. People with an external locus of control believes\
\ fate and luck play an important role in their lives, while people with an internal\
\ locus of control believes they control their lives. The answer is (D)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_psychology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml
View file @ 815f59e6
dataset_name: high_school_statistics
description: 'The following are multiple choice questions (with answers) about high
school statistics.
Q: A new smartwatch is manufactured in one part of a factory, then secured for shipping
in another, independent part of the factory. The weight of the smartwatch has a
mean of 62 grams and a standard deviation of 1.0 grams. The weight of the packaging
(box, user''s guide, bubble wrap, etc.) has a mean of 456 grams and a standard deviation
of 6 grams. Together, the distribution of the weight of the smartwatch and its packaging
would have the following mean and standard deviation:
(A) Mean 518 grams; standard deviation 7.0 grams (B) Mean 518 grams; standard deviation
3.5 grams (C) Mean 518 grams; standard deviation 6.1 grams (D) Mean 394 grams; standard
deviation 6.1 grams
A: Let''s think step by step. Since the weight of the watch and the weight of the
packaging are independent random variables, the mean and variance of their sum is
equal to the sum of their individual means and variances. So the mean is 62 + 456
= 518 grams, and the variances is 1.0^2 + 6.0^2 = 37, leading to a standard deviation
of 6.1 grams. The answer is (C).
Q: After a frost warning was issued, the owner of a large orange grove asked his
workers to spray all his trees with water. The water was supposed to freeze and
form a protective covering of ice around the orange blossom. Nevertheless, the owner
suspected that some trees suffered considerable damage due to the frost. To estimate
the proportion of trees that suffered more than 50 percent damage due to the frost,
he took a random sample of 100 trees from his grove. What is the response variable
in this experiment?
(A) The proportion of trees that suffered more than 50 percent damage due to frost.
(B) The number of trees affected by the frost. (C) The number of trees sampled from
the grove. (D) For each sampled tree, whether it suffered more than 50 percent damage
or at most 50 percent damage.
A: Let''s think step by step. In this experiment, the response variable is what
is measured. For each tree, what is measured is whether or not it suffered more
than 50 percent damage due to the frost. The answer is (D).
Q: Suppose X and Y are random variables with E(X) = 37, var(X) = 5, E(Y) = 62, and
var(Y) = 12. What are the expected value and variance of the random variable X +
Y?
(A) E(X + Y) = 99, var(X + Y) = 8.5 (B) E(X + Y) = 99, var(X + Y) = 13 (C) E(X +
Y) = 99, var(X + Y) = 17 (D) There is insufficient information to answer this question.
A: Let''s think step by step. While means of sums of random variables add (regardless
of whether the variables are independent) in order to determine the variance of
a sum of random variables, we need to know not just their individual variances but
the covariance of the two variables, which is not given in this problem. The answer
is (D).
Q: Which of the following sets has the smallest standard deviation? Which has the
largest?
I: {1,2,3}
II: {-10,10}
III: {100}
(A) I, II (B) II, III (C) III, I (D) III, II
A: Let''s think step by step. The variance of distribution I is the expected squared
deviation from its mean (which is 2), so the variance is 2/3 . The variance of distribution
II is 10^2 (because both elements are 10 away from the mean of zero). The variance
of distribution III is 0, since it has a single entry. So distribution III has the
smallest standard deviation and distribution II has the largest. The answer is (D).
Q: Which of the following is a correct statement about correlation?
(A) If the slope of the regression line is exactly 1, then the correlation is exactly
1. (B) If the correlation is 0, then the slope of the regression line is undefined.
(C) Switching which variable is called x and which is called y changes the sign
of the correlation. (D) The correlation r is equal to the slope of the regression
line when z-scores for the y-variable are plotted against z-scores for the x-variable.
A: Let''s think step by step. Statement A is false because the slope of the regression
line being exactly 1 can occur even when the two variables are not perfectly correlated.
Statement B is false because uncorrelated variables regression lines can have slope
zero. Statement C is false because correlation is symmetric in the two random variables.
The answer is (D).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_statistics
"dataset_name": "high_school_statistics"
"description": "The following are multiple choice questions (with answers) about high\
\ school statistics.\n\nQ: A new smartwatch is manufactured in one part of a factory,\
\ then secured for shipping in another, independent part of the factory. The weight\
\ of the smartwatch has a mean of 62 grams and a standard deviation of 1.0 grams.\
\ The weight of the packaging (box, user's guide, bubble wrap, etc.) has a mean\
\ of 456 grams and a standard deviation of 6 grams. Together, the distribution of\
\ the weight of the smartwatch and its packaging would have the following mean and\
\ standard deviation:\n(A) Mean 518 grams; standard deviation 7.0 grams (B) Mean\
\ 518 grams; standard deviation 3.5 grams (C) Mean 518 grams; standard deviation\
\ 6.1 grams (D) Mean 394 grams; standard deviation 6.1 grams\nA: Let's think step\
\ by step. Since the weight of the watch and the weight of the packaging are independent\
\ random variables, the mean and variance of their sum is equal to the sum of their\
\ individual means and variances. So the mean is 62 + 456 = 518 grams, and the variances\
\ is 1.0^2 + 6.0^2 = 37, leading to a standard deviation of 6.1 grams. The answer\
\ is (C).\n\nQ: After a frost warning was issued, the owner of a large orange grove\
\ asked his workers to spray all his trees with water. The water was supposed to\
\ freeze and form a protective covering of ice around the orange blossom. Nevertheless,\
\ the owner suspected that some trees suffered considerable damage due to the frost.\
\ To estimate the proportion of trees that suffered more than 50 percent damage\
\ due to the frost, he took a random sample of 100 trees from his grove. What is\
\ the response variable in this experiment?\n(A) The proportion of trees that suffered\
\ more than 50 percent damage due to frost. (B) The number of trees affected by\
\ the frost. (C) The number of trees sampled from the grove. (D) For each sampled\
\ tree, whether it suffered more than 50 percent damage or at most 50 percent damage.\n\
A: Let's think step by step. In this experiment, the response variable is what is\
\ measured. For each tree, what is measured is whether or not it suffered more than\
\ 50 percent damage due to the frost. The answer is (D).\n\nQ: Suppose X and Y are\
\ random variables with E(X) = 37, var(X) = 5, E(Y) = 62, and var(Y) = 12. What\
\ are the expected value and variance of the random variable X + Y?\n(A) E(X + Y)\
\ = 99, var(X + Y) = 8.5 (B) E(X + Y) = 99, var(X + Y) = 13 (C) E(X + Y) = 99, var(X\
\ + Y) = 17 (D) There is insufficient information to answer this question.\nA: Let's\
\ think step by step. While means of sums of random variables add (regardless of\
\ whether the variables are independent) in order to determine the variance of a\
\ sum of random variables, we need to know not just their individual variances but\
\ the covariance of the two variables, which is not given in this problem. The answer\
\ is (D).\n\nQ: Which of the following sets has the smallest standard deviation?\
\ Which has the largest?\nI: {1,2,3}\nII: {-10,10}\nIII: {100}\n(A) I, II (B) II,\
\ III (C) III, I (D) III, II\nA: Let's think step by step. The variance of distribution\
\ I is the expected squared deviation from its mean (which is 2), so the variance\
\ is 2/3 . The variance of distribution II is 10^2 (because both elements are 10\
\ away from the mean of zero). The variance of distribution III is 0, since it has\
\ a single entry. So distribution III has the smallest standard deviation and distribution\
\ II has the largest. The answer is (D).\n\nQ: Which of the following is a correct\
\ statement about correlation?\n(A) If the slope of the regression line is exactly\
\ 1, then the correlation is exactly 1. (B) If the correlation is 0, then the slope\
\ of the regression line is undefined. (C) Switching which variable is called x\
\ and which is called y changes the sign of the correlation. (D) The correlation\
\ r is equal to the slope of the regression line when z-scores for the y-variable\
\ are plotted against z-scores for the x-variable.\nA: Let's think step by step.\
\ Statement A is false because the slope of the regression line being exactly 1\
\ can occur even when the two variables are not perfectly correlated. Statement\
\ B is false because uncorrelated variables regression lines can have slope zero.\
\ Statement C is false because correlation is symmetric in the two random variables.\
\ The answer is (D)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_statistics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml
View file @ 815f59e6
dataset_name: high_school_world_history
description: "The following are multiple choice questions (with answers) about high\
"dataset_name": "high_school_world_history"
"description": "The following are multiple choice questions (with answers) about high\
\ school world history.\n\nQ: This question refers to the following information.\n\
\"At least one of the [world's] societies would have to somehow enormously increase\
\ its productivity [in order to achieve global hegemony]. That quantum jump would\
......@@ -8,75 +8,75 @@ description: "The following are multiple choice questions (with answers) about h
\ be accomplished by exploiting the ecosystems, mineral resources, and human assets\
\ of whole continents outside the lands of the society making the jump. Western\
\ Europe did just that by means of its brutality and guns and, more important, by\
\ geographical and ecological luck.\"\nCopyright \xA9 2015 Cambridge University\
\ Press.\nAlfred Crosby, historian, Ecological Imperialism, 2004\nThe \"quantum\
\ jump\" mentioned in the passage most directly contributed to which of the following\
\ developments in the period 1450\u20131750 C.E.?\n(A) A breakdown in trade routes\
\ through the collapse of the established state structure (B) An increase in the\
\ population of the world through more plentiful supplies of food (C) The spread\
\ of Chinese and Indian belief systems across the world (D) An increase in social\
\ unrest\nA: Let's think step by step. We refer to Wikipedia articles on world history\
\ for help. The \"quantum jump\" mentioned in the passage refers to the conquest\
\ of the New World and the Columbian Exchange. Choice (A) and (C) did not happen\
\ in history. Choice (C) refers to the human assets. The answer is (B).\n\nQ: This\
\ question refers to the following information.\n\"The struggle against neo-colonialism\
\ is not aimed at excluding the capital of the developed world from operating in\
\ less developed countries. It is aimed at preventing the financial power of the\
\ developed countries being used in such a way as to impoverish the less developed.\n\
Non-alignment, as practiced by Ghana and many other countries, is based on co-operation\
\ with all States whether they be capitalist, socialist or have a mixed economy.\
\ Such a policy, therefore, involves foreign investment from capitalist countries,\
\ but it must be invested in accordance with a national plan drawn up by the government\
\ of the non-aligned State with its own interests in mind. The issue is not what\
\ return the foreign investor receives on his investments\u2026The question is one\
\ of power. A State in the grip of neo-colonialism is not master of its own destiny.\"\
\nKwame Nkrumah, Neo-Colonialism, 1965\nWhich of the following provides the best\
\ context for Nkrumah's writings?\n(A) The Industrial Revolution (B) Decolonization\
\ (C) Regional Free Trade Associations (D) Autarky\nA: Let's think step by step.\
\ We refer to Wikipedia articles on world history for help. The passage expresses\
\ a point that the successful fight against neo-colonialism were in danger and the\
\ newly independent nations like Ghana may be re-colonized via financial power of\
\ the developed countries. The answer is (B).\n\nQ: This question refers to the\
\ following information.\n\"Indeed, as both the fatwas of distinguished [scholars]\
\ who base their opinion on reason and tradition alike and the consensus of the\
\ Sunni community agree that the ancient obligation of extirpation, extermination,\
\ and expulsion of evil innovation must be the aim of our exalted aspiration, for\
\ \"Religious zeal is a victory for the Faith of God the Beneficent\"; then, in\
\ accordance with the words of the Prophet (Peace upon him!) \"Whosoever introduces\
\ evil innovation into our order must be expelled\" and \"Whosoever does aught against\
\ our order must be expelled,\" action has become necessary and exigent\u2026\"\n\
Letter from Ottoman Sultan Selim I to Safavid Shah Ismail I, 1514\nThe letter from\
\ Selim I is most clearly an example of which of the following?\n(A) The maintenance\
\ of military supremacy at all costs (B) Expanding tensions between religious sects\
\ (C) Factors that brought about the collapse of the Ottoman Empire (D) Peacemaking\
\ efforts among the Islamic empires\nA: Let's think step by step. We refer to Wikipedia\
\ articles on world history for help. The passage is an example of expanding tensions\
\ between Selim and Ismail. In the passage the Selim references the fatwa and the\
\ consensus of the Sunni community to against whosoever introduces evil. The answer\
\ is (B).\n\nQ: This question refers to the following information.\n\"The real grievance\
\ of the worker is the insecurity of his existence; he is not sure that he will\
\ always have work, he is not sure that he will always be healthy, and he foresees\
\ that he will one day be old and unfit to work. If he falls into poverty, even\
\ if only through a prolonged illness, he is then completely helpless, exam_ins\
\ to his own devices, and society does not currently recognize any real obligation\
\ towards him beyond the usual help for the poor, even if he has been working all\
\ the time ever so faithfully and diligently. The usual help for the poor, however,\
\ leaves a lot to be desired, especially in large cities, where it is very much\
\ worse than in the country.\"\nOtto von Bismarck, 1884\nOtto von Bismarck likely\
\ made this speech in reaction to which of the following issues?\n(A) Social acceptance\
\ of child labor (B) Declining life expectancy in Germany (C) Criticisms of German\
\ trade tariffs (D) Negative effects attributed to industrial capitalism\nA: Let's\
\ think step by step. We refer to Wikipedia articles on world history for help.\
\ The passage talks about the grievance of the work under the industrial capitalism.\
\ The answer is (D).\n\nQ: This question refers to the following information.\n\
He contains all works and desires and all perfumes and all tastes. He enfolds the\
\ whole universe and in silence is loving to all. This is the Spirit that is in\
\ my heart, this is Brahman. To him I shall come when I go beyond this life, and\
\ to him will come he who has faith and doubts not.\n\u2014The Upanishads, India,\
\ c. 1000 BCE\nTo which religion does the speaker most likely belong?\n(A) Hinduism\
\ (B) Buddhism (C) Shintoism (D) Zoroastrianism\nA: Let's think step by step. We\
\ refer to Wikipedia articles on world history for help. Brahman refers to the ultimate\
\ reality of all things in the Hindu religion. In contrast, Buddhism does not have\
\ a concept of supreme God. The answer is (A)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_world_history
\ geographical and ecological luck.\"\nCopyright © 2015 Cambridge University Press.\n\
Alfred Crosby, historian, Ecological Imperialism, 2004\nThe \"quantum jump\" mentioned\
\ in the passage most directly contributed to which of the following developments\
\ in the period 1450–1750 C.E.?\n(A) A breakdown in trade routes through the collapse\
\ of the established state structure (B) An increase in the population of the world\
\ through more plentiful supplies of food (C) The spread of Chinese and Indian belief\
\ systems across the world (D) An increase in social unrest\nA: Let's think step\
\ by step. We refer to Wikipedia articles on world history for help. The \"quantum\
\ jump\" mentioned in the passage refers to the conquest of the New World and the\
\ Columbian Exchange. Choice (A) and (C) did not happen in history. Choice (C) refers\
\ to the human assets. The answer is (B).\n\nQ: This question refers to the following\
\ information.\n\"The struggle against neo-colonialism is not aimed at excluding\
\ the capital of the developed world from operating in less developed countries.\
\ It is aimed at preventing the financial power of the developed countries being\
\ used in such a way as to impoverish the less developed.\nNon-alignment, as practiced\
\ by Ghana and many other countries, is based on co-operation with all States whether\
\ they be capitalist, socialist or have a mixed economy. Such a policy, therefore,\
\ involves foreign investment from capitalist countries, but it must be invested\
\ in accordance with a national plan drawn up by the government of the non-aligned\
\ State with its own interests in mind. The issue is not what return the foreign\
\ investor receives on his investments…The question is one of power. A State in\
\ the grip of neo-colonialism is not master of its own destiny.\"\nKwame Nkrumah,\
\ Neo-Colonialism, 1965\nWhich of the following provides the best context for Nkrumah's\
\ writings?\n(A) The Industrial Revolution (B) Decolonization (C) Regional Free\
\ Trade Associations (D) Autarky\nA: Let's think step by step. We refer to Wikipedia\
\ articles on world history for help. The passage expresses a point that the successful\
\ fight against neo-colonialism were in danger and the newly independent nations\
\ like Ghana may be re-colonized via financial power of the developed countries.\
\ The answer is (B).\n\nQ: This question refers to the following information.\n\"\
Indeed, as both the fatwas of distinguished [scholars] who base their opinion on\
\ reason and tradition alike and the consensus of the Sunni community agree that\
\ the ancient obligation of extirpation, extermination, and expulsion of evil innovation\
\ must be the aim of our exalted aspiration, for \"Religious zeal is a victory for\
\ the Faith of God the Beneficent\"; then, in accordance with the words of the Prophet\
\ (Peace upon him!) \"Whosoever introduces evil innovation into our order must be\
\ expelled\" and \"Whosoever does aught against our order must be expelled,\" action\
\ has become necessary and exigent…\"\nLetter from Ottoman Sultan Selim I to Safavid\
\ Shah Ismail I, 1514\nThe letter from Selim I is most clearly an example of which\
\ of the following?\n(A) The maintenance of military supremacy at all costs (B)\
\ Expanding tensions between religious sects (C) Factors that brought about the\
\ collapse of the Ottoman Empire (D) Peacemaking efforts among the Islamic empires\n\
A: Let's think step by step. We refer to Wikipedia articles on world history for\
\ help. The passage is an example of expanding tensions between Selim and Ismail.\
\ In the passage the Selim references the fatwa and the consensus of the Sunni community\
\ to against whosoever introduces evil. The answer is (B).\n\nQ: This question refers\
\ to the following information.\n\"The real grievance of the worker is the insecurity\
\ of his existence; he is not sure that he will always have work, he is not sure\
\ that he will always be healthy, and he foresees that he will one day be old and\
\ unfit to work. If he falls into poverty, even if only through a prolonged illness,\
\ he is then completely helpless, exam_ins to his own devices, and society does\
\ not currently recognize any real obligation towards him beyond the usual help\
\ for the poor, even if he has been working all the time ever so faithfully and\
\ diligently. The usual help for the poor, however, leaves a lot to be desired,\
\ especially in large cities, where it is very much worse than in the country.\"\
\nOtto von Bismarck, 1884\nOtto von Bismarck likely made this speech in reaction\
\ to which of the following issues?\n(A) Social acceptance of child labor (B) Declining\
\ life expectancy in Germany (C) Criticisms of German trade tariffs (D) Negative\
\ effects attributed to industrial capitalism\nA: Let's think step by step. We refer\
\ to Wikipedia articles on world history for help. The passage talks about the grievance\
\ of the work under the industrial capitalism. The answer is (D).\n\nQ: This question\
\ refers to the following information.\nHe contains all works and desires and all\
\ perfumes and all tastes. He enfolds the whole universe and in silence is loving\
\ to all. This is the Spirit that is in my heart, this is Brahman. To him I shall\
\ come when I go beyond this life, and to him will come he who has faith and doubts\
\ not.\n—The Upanishads, India, c. 1000 BCE\nTo which religion does the speaker\
\ most likely belong?\n(A) Hinduism (B) Buddhism (C) Shintoism (D) Zoroastrianism\n\
A: Let's think step by step. We refer to Wikipedia articles on world history for\
\ help. Brahman refers to the ultimate reality of all things in the Hindu religion.\
\ In contrast, Buddhism does not have a concept of supreme God. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_world_history"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml
View file @ 815f59e6
dataset_name: human_aging
description: 'The following are multiple choice questions (with answers) about human
aging.
Q: All other things being equal, which of the following persons is more likely to
show osteoporosis?
(A) An older Hispanic American woman (B) An older African American woman (C) An
older Asian American woman (D) An older Native American woman
A: Let''s think step by step. We refer to Wikipedia articles on human aging for
help. Although osteoporosis can occur at any age, the risk is higher for older people.
It is most common in Asian and non-Hispanic white women. The answer is (C).
Q: The finding that adults tend to remember events from their adolescence better
than from other periods in their lives is referred to as the
(A) Adolescence advantage (B) Reminiscence bump (C) Memorial memorial (D) Quadratic
retrieval spike
A: Let''s think step by step. We refer to Wikipedia articles on human aging for
help. Reminiscence bump is a phenomenon that older adults tend to recollect events
during their young ages. People usually have a period of childhood amnesia from
birth to around age 5, and a reminiscence bump between 10 and 30. The answer is
(B).
Q: Which element in tobacco smoke is responsible for cancers?
(A) Nicotine (B) Tar (C) Carbon monoxide (D) Smoke particles
A: Let''s think step by step. We refer to Wikipedia articles on human aging for
help. The benzene, acrylamide and acrylonitrile in tar interact with the lungs and
cause DNA mutations in cells of the lungs, and lead to cancer. The answer is (B).
Q: When older adults move to a new state after retirement, which of the following
is the more likely destination?
(A) Texas (B) California (C) Hawaii (D) Vermont
A: Let''s think step by step. We refer to Wikipedia articles on human aging for
help. Texas does not have state tax, and has low cost of living compared with the
other three options. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_human_aging
"dataset_name": "human_aging"
"description": "The following are multiple choice questions (with answers) about human\
\ aging.\n\nQ: All other things being equal, which of the following persons is more\
\ likely to show osteoporosis?\n(A) An older Hispanic American woman (B) An older\
\ African American woman (C) An older Asian American woman (D) An older Native American\
\ woman\nA: Let's think step by step. We refer to Wikipedia articles on human aging\
\ for help. Although osteoporosis can occur at any age, the risk is higher for older\
\ people. It is most common in Asian and non-Hispanic white women. The answer is\
\ (C).\n\nQ: The finding that adults tend to remember events from their adolescence\
\ better than from other periods in their lives is referred to as the\n(A) Adolescence\
\ advantage (B) Reminiscence bump (C) Memorial memorial (D) Quadratic retrieval\
\ spike\nA: Let's think step by step. We refer to Wikipedia articles on human aging\
\ for help. Reminiscence bump is a phenomenon that older adults tend to recollect\
\ events during their young ages. People usually have a period of childhood amnesia\
\ from birth to around age 5, and a reminiscence bump between 10 and 30. The answer\
\ is (B).\n\nQ: Which element in tobacco smoke is responsible for cancers?\n(A)\
\ Nicotine (B) Tar (C) Carbon monoxide (D) Smoke particles\nA: Let's think step\
\ by step. We refer to Wikipedia articles on human aging for help. The benzene,\
\ acrylamide and acrylonitrile in tar interact with the lungs and cause DNA mutations\
\ in cells of the lungs, and lead to cancer. The answer is (B).\n\nQ: When older\
\ adults move to a new state after retirement, which of the following is the more\
\ likely destination?\n(A) Texas (B) California (C) Hawaii (D) Vermont\nA: Let's\
\ think step by step. We refer to Wikipedia articles on human aging for help. Texas\
\ does not have state tax, and has low cost of living compared with the other three\
\ options. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_human_aging"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml
View file @ 815f59e6
dataset_name: human_sexuality
description: 'The following are multiple choice questions (with answers) about human
sexuality.
Q: The most common disorder among men who seek sexual therapy is:
(A) premature ejaculation (B) inhibited ejaculation (C) erectile disorder (D) ejaculatory
disorder
A: Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. The most common type of arousal disorder for men is erectile dysfunction,
meaning that a person is not able to get any physical satisfaction from sexual activity
although he may be interested in it. The answer is (C).
Q: A woman who knows she has active herpes and untreated syphilis but continues
to have sex without informing her partners of her condition has, in psychoanalytic
terms:
(A) a strong ego (B) a weak superego (C) a weak id (D) a strong superego
A: Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. A person with weak superego tends to be delinquent, criminal or have antisocial
personality. The action of the woman who knows she has active venereal disease but
still have sex with her partners indicate she may has antisocial personality. The
answer is (B).
Q: The nature of homosexual activities that occur during preadolescence include
all but which of the following?
(A) sexual intercourse (B) circle jerks (C) exhibitionism (D) touching each other''s
genitals
A: Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. Sexual intercourse is prohibited during preadolescence. The answer is
(A).
Q: Women''s ability to have multiple orgasms is primarily due to:
(A) the fact that they do not have a refractory period. (B) the response of the
inner layers of the vagina. (C) having alternating orgasms in different locations.
(D) the G-Spot.
A: Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. The refractory period is the time when a person is not able to be erect
or is not interested in sex. The answer is (A).
Q: Morning sickness is typically a problem:
(A) during the first trimester (B) during the second trimester (C) during the third
trimester (D) all through the pregnancy
A: Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. Morning sickness usually begins by nine weeks after conception, corresponding
to the first trimester. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_human_sexuality
"dataset_name": "human_sexuality"
"description": "The following are multiple choice questions (with answers) about human\
\ sexuality.\n\nQ: The most common disorder among men who seek sexual therapy is:\n\
(A) premature ejaculation (B) inhibited ejaculation (C) erectile disorder (D) ejaculatory\
\ disorder\nA: Let's think step by step. We refer to Wikipedia articles on human\
\ sexuality for help. The most common type of arousal disorder for men is erectile\
\ dysfunction, meaning that a person is not able to get any physical satisfaction\
\ from sexual activity although he may be interested in it. The answer is (C).\n\
\nQ: A woman who knows she has active herpes and untreated syphilis but continues\
\ to have sex without informing her partners of her condition has, in psychoanalytic\
\ terms:\n(A) a strong ego (B) a weak superego (C) a weak id (D) a strong superego\n\
A: Let's think step by step. We refer to Wikipedia articles on human sexuality for\
\ help. A person with weak superego tends to be delinquent, criminal or have antisocial\
\ personality. The action of the woman who knows she has active venereal disease\
\ but still have sex with her partners indicate she may has antisocial personality.\
\ The answer is (B).\n\nQ: The nature of homosexual activities that occur during\
\ preadolescence include all but which of the following?\n(A) sexual intercourse\
\ (B) circle jerks (C) exhibitionism (D) touching each other's genitals\nA: Let's\
\ think step by step. We refer to Wikipedia articles on human sexuality for help.\
\ Sexual intercourse is prohibited during preadolescence. The answer is (A).\n\n\
Q: Women's ability to have multiple orgasms is primarily due to:\n(A) the fact that\
\ they do not have a refractory period. (B) the response of the inner layers of\
\ the vagina. (C) having alternating orgasms in different locations. (D) the G-Spot.\n\
A: Let's think step by step. We refer to Wikipedia articles on human sexuality for\
\ help. The refractory period is the time when a person is not able to be erect\
\ or is not interested in sex. The answer is (A).\n\nQ: Morning sickness is typically\
\ a problem:\n(A) during the first trimester (B) during the second trimester (C)\
\ during the third trimester (D) all through the pregnancy\nA: Let's think step\
\ by step. We refer to Wikipedia articles on human sexuality for help. Morning sickness\
\ usually begins by nine weeks after conception, corresponding to the first trimester.\
\ The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_human_sexuality"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml
View file @ 815f59e6
dataset_name: international_law
description: 'The following are multiple choice questions (with answers) about international
law.
Q: How the consent to be bound of a State may be expressed?
(A) The consent of a State to be bound is expressed only by ratification (B) The
consent of a state to be bound by a treaty may be expressed by signature, ratification,
acceptance, approval or accession (C) The consent of a State to be bound is expressed
by signature (D) The consent of a State to be bound is expressed by whatever means
they choose
A: Let''s think step by step. We refer to Wikipedia articles on international law
for help. Article 11 of Vienna Convention on the Law of Treaties signed in 1969
states that "the consent of a State to be bound by a treaty may be expressed by
signature, exchange of instruments constituting a treaty, ratification, acceptance,
approval or accession, or by any other means if so agreed." (B) is the most precise
and accurate answer. The answer is (B).
Q: What is the judge ad hoc?
(A) If a party to a contentious case before the ICJ does not have a national sitting
as judge, it is entitled to nominate someone as a judge solely for that case, with
the title of judge ad hoc (B) Judge ad hoc is the member of the bench of the ICJ
with a casting vote (C) Judge ad hoc is a surrogate judge, in case a judge is disqualified
or passes away (D) Judge ad hoc is the judge that each party will always nominate
in every contentious case
A: Let''s think step by step. We refer to Wikipedia articles on international law
for help. As "ad hoc" implies, a judge ad hoc is appointed only for a specific case
or period, when a party to a contentious case before the International Court of
Justice does not have a regular national sitting as judge. The answer is (A).
Q: When ''consent'' can serve as a circumstance precluding the wrongfulness of a
State conduct?
(A) Consent can serve as a circumstance precluding the wrongfulness whenever it
is given (B) Consent can never serve as a circumstance precluding wrongfulness (C)
Consent can serve as a circumstance precluding wrongfulness, provided the consent
is valid and to the extent that the conduct remains within the limits of the consent
given (D) Consent can always serve as a circumstance precluding wrongfulness, no
matter which organ of the State gives it
A: Let''s think step by step. We refer to Wikipedia articles on international law
for help. Valid consent can serve as a circumstance precluding the wrongfulness
of a State conduct if the conduct remains within the limits of that consent, according
to Chapter V of the Responsibility of States for Internationally Wrongful Acts,
2001, United Nations. The answer is (C).
Q: Would a reservation to the definition of torture in the ICCPR be acceptable in
contemporary practice?
(A) This is an acceptable reservation if the reserving country''s legislation employs
a different definition (B) This is an unacceptable reservation because it contravenes
the object and purpose of the ICCPR (C) This is an unacceptable reservation because
the definition of torture in the ICCPR is consistent with customary international
law (D) This is an acceptable reservation because under general international law
States have the right to enter reservations to treaties
A: Let''s think step by step. We refer to Wikipedia articles on international law
for help. For it contravenes the object and purpose of the ICCPR, this is an unacceptable
reservation in contemporary practice. The answer is (B).
Q: What types of force does Article 2(4) of the UN Charter prohibit?
(A) Article 2(4) encompasses only armed force (B) Article 2(4) encompasses all types
of force, including sanctions (C) Article 2(4) encompasses all interference in the
domestic affairs of States (D) Article 2(4) encompasses force directed only against
a State''s territorial integrity
A: Let''s think step by step. We refer to Wikipedia articles on international law
for help. Article 2(4) of the UN Charter prohibits states from using armed forces
in their international relations. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_international_law
"dataset_name": "international_law"
"description": "The following are multiple choice questions (with answers) about international\
\ law.\n\nQ: How the consent to be bound of a State may be expressed?\n(A) The consent\
\ of a State to be bound is expressed only by ratification (B) The consent of a\
\ state to be bound by a treaty may be expressed by signature, ratification, acceptance,\
\ approval or accession (C) The consent of a State to be bound is expressed by signature\
\ (D) The consent of a State to be bound is expressed by whatever means they choose\n\
A: Let's think step by step. We refer to Wikipedia articles on international law\
\ for help. Article 11 of Vienna Convention on the Law of Treaties signed in 1969\
\ states that \"the consent of a State to be bound by a treaty may be expressed\
\ by signature, exchange of instruments constituting a treaty, ratification, acceptance,\
\ approval or accession, or by any other means if so agreed.\" (B) is the most precise\
\ and accurate answer. The answer is (B).\n\nQ: What is the judge ad hoc?\n(A) If\
\ a party to a contentious case before the ICJ does not have a national sitting\
\ as judge, it is entitled to nominate someone as a judge solely for that case,\
\ with the title of judge ad hoc (B) Judge ad hoc is the member of the bench of\
\ the ICJ with a casting vote (C) Judge ad hoc is a surrogate judge, in case a judge\
\ is disqualified or passes away (D) Judge ad hoc is the judge that each party will\
\ always nominate in every contentious case\nA: Let's think step by step. We refer\
\ to Wikipedia articles on international law for help. As \"ad hoc\" implies, a\
\ judge ad hoc is appointed only for a specific case or period, when a party to\
\ a contentious case before the International Court of Justice does not have a regular\
\ national sitting as judge. The answer is (A).\n\nQ: When 'consent' can serve as\
\ a circumstance precluding the wrongfulness of a State conduct?\n(A) Consent can\
\ serve as a circumstance precluding the wrongfulness whenever it is given (B) Consent\
\ can never serve as a circumstance precluding wrongfulness (C) Consent can serve\
\ as a circumstance precluding wrongfulness, provided the consent is valid and to\
\ the extent that the conduct remains within the limits of the consent given (D)\
\ Consent can always serve as a circumstance precluding wrongfulness, no matter\
\ which organ of the State gives it\nA: Let's think step by step. We refer to Wikipedia\
\ articles on international law for help. Valid consent can serve as a circumstance\
\ precluding the wrongfulness of a State conduct if the conduct remains within the\
\ limits of that consent, according to Chapter V of the Responsibility of States\
\ for Internationally Wrongful Acts, 2001, United Nations. The answer is (C).\n\n\
Q: Would a reservation to the definition of torture in the ICCPR be acceptable in\
\ contemporary practice?\n(A) This is an acceptable reservation if the reserving\
\ country's legislation employs a different definition (B) This is an unacceptable\
\ reservation because it contravenes the object and purpose of the ICCPR (C) This\
\ is an unacceptable reservation because the definition of torture in the ICCPR\
\ is consistent with customary international law (D) This is an acceptable reservation\
\ because under general international law States have the right to enter reservations\
\ to treaties\nA: Let's think step by step. We refer to Wikipedia articles on international\
\ law for help. For it contravenes the object and purpose of the ICCPR, this is\
\ an unacceptable reservation in contemporary practice. The answer is (B).\n\nQ:\
\ What types of force does Article 2(4) of the UN Charter prohibit?\n(A) Article\
\ 2(4) encompasses only armed force (B) Article 2(4) encompasses all types of force,\
\ including sanctions (C) Article 2(4) encompasses all interference in the domestic\
\ affairs of States (D) Article 2(4) encompasses force directed only against a State's\
\ territorial integrity\nA: Let's think step by step. We refer to Wikipedia articles\
\ on international law for help. Article 2(4) of the UN Charter prohibits states\
\ from using armed forces in their international relations. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_international_law"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment