Skip to content

GitLab

Unverified Commit a2af2101 authored by Yen-Ting Lin's avatar Yen-Ting Lin Committed by GitHub
Browse files

Merge branch 'EleutherAI:main' into main

parents 82cb25c1 d5f39bf8
Hide whitespace changes
Inline Side-by-side
Showing with 1483 additions and 1147 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml
View file @ a2af2101
"dataset_name": "formal_logic" dataset_name: formal_logic
"description": "The following are multiple choice questions (with answers) about formal\ description: The following are multiple choice questions (with answers) about formal
\ logic.\n\nQ: Which of the given formulas of PL is the best symbolization of the\ logic.
\ following sentence?\nTurtles live long lives and are happy creatures, unless they\ fewshot_config:
\ are injured.\n(A) (L • H) ≡ I (B) (L • H) ∨ I (C) L • (H ∨ I) (D) L • (H ⊃ R).\n\ sampler: first_n
A: Let's think step by step. We refer to Wikipedia articles on formal logic for\ samples:
\ help. Let’s solve this step by step. Let “L” denote “living long”, H “being happy”,\ - question: "Which of the given formulas of PL is the best symbolization of the following\
\ and “I” “being injured”. Now, consider each choice:\n(A) means (living long AND\ \ sentence?\nTurtles live long lives and are happy creatures, unless they are\
\ being happy) is equivalent to (being injured). \n(B) means (living long AND being\ \ injured.\n(A) (L \u2022 H) \u2261 I (B) (L \u2022 H) \u2228 I (C) L \u2022\
\ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\ \ (H \u2228 I) (D) L \u2022 (H \u2283 R)."
\ injured). \n(D) means (living long) AND (being happy implies being R), but what\ target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
\ R denotes is not clear.\nObviously, (B) is the best symbolization of the original\ \ for help. Let\u2019s solve this step by step. Let \u201CL\u201D denote \u201C\
\ sentence. The answer is (B).\n\nQ: Select the best translation into predicate\ living long\u201D, H \u201Cbeing happy\u201D, and \u201CI\u201D \u201Cbeing\
\ logic.George borrows Hector's lawnmower. (g: George; h: Hector; l: Hector's lawnmower;\ \ injured\u201D. Now, consider each choice:\n(A) means (living long AND being\
\ Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh (D) Bghl\nA: Let's think\ \ happy) is equivalent to (being injured). \n(B) means (living long AND being\
\ step by step. We refer to Wikipedia articles on formal logic for help. Let’s solve\ \ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\
\ this step by step. We are told that “Bxyx” means “x borrows y from z”. We can\ \ injured). \n(D) means (living long) AND (being happy implies being R), but\
\ rewrite “George borrows Hector's lawnmower” as “George borrows a lawnmower from\ \ what R denotes is not clear.\nObviously, (B) is the best symbolization of\
\ Hector”, which can then be translated into predicate logic as “Bglh”. The answer\ \ the original sentence. The answer is (B)."
\ “Bglh” appears in (C); therefore, (C) must be the correct answer. The answer is\ - question: 'Select the best translation into predicate logic.George borrows Hector''s
\ (C).\n\nQ: \nSelect the best English interpretation of the given arguments in\ lawnmower. (g: George; h: Hector; l: Hector''s lawnmower; Bxyx: x borrows y
\ predicate logic.\nDm\n(∀x)(Wx ⊃ ~Dx). \n(∀x)Wx ∨ Ag\t/ (∃x)Ax\n(A) Marina is a\ from z).
\ dancer. Some weaklings are not dancers. Either everything is a weakling or Georgia\
\ plays volleyball. So something plays volleyball. (B) Marina is a dancer. No weakling\ (A) Blgh (B) Bhlg (C) Bglh (D) Bghl'
\ is a dancer. Everything is either a weakling or plays volleyball. So something\ target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
\ plays volleyball. (C) Marina is a dancer. Some weaklings are not dancers. Everything\ \ for help. Let\u2019s solve this step by step. We are told that \u201CBxyx\u201D\
\ is either a weakling or plays volleyball. So something plays volleyball. (D) Marina\ \ means \u201Cx borrows y from z\u201D. We can rewrite \u201CGeorge borrows\
\ is a dancer. No weakling is a dancer. Either everything is a weakling or Georgia\ \ Hector's lawnmower\u201D as \u201CGeorge borrows a lawnmower from Hector\u201D\
\ plays volleyball. So something plays volleyball.\nA: Let's think step by step.\ , which can then be translated into predicate logic as \u201CBglh\u201D. The\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\ \ answer \u201CBglh\u201D appears in (C); therefore, (C) must be the correct\
\ by step. Let “D” denote “being a dancer”, “m” denote “Maria”, “g” denote “Georgia”,\ \ answer. The answer is (C)."
\ “W” denote “weakling”, “A” denote “playing volleyball”. Then, we have the following:\n\ - question: "\nSelect the best English interpretation of the given arguments in predicate\
1. Dm → Maria is a dance.\n2. (∀x)(Wx ⊃ ~Dx). → For all x, if x is a weakling, then\ \ logic.\nDm\n(\u2200x)(Wx \u2283 ~Dx). \n(\u2200x)Wx \u2228 Ag\t/ (\u2203x)Ax\n\
\ x is not a dancer. In other words, no weakling is a dancer.\n3. (∀x)Wx ∨ Ag\t\ (A) Marina is a dancer. Some weaklings are not dancers. Either everything is\
/ (∃x)Ax → For all x, x is a weakling or Georgia plays volleyball. So there exists\ \ a weakling or Georgia plays volleyball. So something plays volleyball. (B)\
\ an x that plays volleyball. \nOptions (A) and (C) do claim that some weaklings\ \ Marina is a dancer. No weakling is a dancer. Everything is either a weakling\
\ are not dancers, but the second argument strongly states that no weakling is a\ \ or plays volleyball. So something plays volleyball. (C) Marina is a dancer.\
\ dancer. Thus, we can eliminate them. Option (B) omits the important detail about\ \ Some weaklings are not dancers. Everything is either a weakling or plays volleyball.\
\ Georgia playing volleyball. Option (D) has all the details presented in the arguments\ \ So something plays volleyball. (D) Marina is a dancer. No weakling is a dancer.\
\ and is the best English interpretation of the arguments. The answer is (D).\n\n\ \ Either everything is a weakling or Georgia plays volleyball. So something\
Q: Select the best translation into predicate logic: No people drive on Mars.\n\ \ plays volleyball."
(A) ~Pd (B) (∀x)(Px ∨ ~Dx) (C) (∀x)(Px ⊃ ~Dx) (D) ~Dp\nA: Let's think step by step.\ target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\ \ for help. Let\u2019s solve this step by step. Let \u201CD\u201D denote \u201C\
\ by step. Let “P” denote “being on Mars” and “D” denote “driving on Mars”. Then\ being a dancer\u201D, \u201Cm\u201D denote \u201CMaria\u201D, \u201Cg\u201D\
\ let’s consider each option:\nOption (A): ~Pd → d is not on Mars.\nOption (B):\ \ denote \u201CGeorgia\u201D, \u201CW\u201D denote \u201Cweakling\u201D, \u201C\
\ (∀x)(Px ∨ ~Dx) → For all x, x is on Mars and x do not drive on Mars.\nOption (C):\ A\u201D denote \u201Cplaying volleyball\u201D. Then, we have the following:\n\
\ (∀x)(Px ⊃ ~Dx) → For all x, x is on Mars implies that x do not drive on Mars.\n\ 1. Dm \u2192 Maria is a dance.\n2. (\u2200x)(Wx \u2283 ~Dx). \u2192 For all\
Option (D): ~Dp: → p do not drive on Mars.\nOf all these options, Option (C) appears\ \ x, if x is a weakling, then x is not a dancer. In other words, no weakling\
\ to be the best and most meaningful interpretation of the argument “No people drive\ \ is a dancer.\n3. (\u2200x)Wx \u2228 Ag\t/ (\u2203x)Ax \u2192 For all x, x\
\ on Mars.” The answer is (C).\n\n" \ is a weakling or Georgia plays volleyball. So there exists an x that plays\
"group": "mmlu_flan_cot_fewshot_humanities" \ volleyball. \nOptions (A) and (C) do claim that some weaklings are not dancers,\
"include": "_mmlu_flan_cot_fewshot_template_yaml" \ but the second argument strongly states that no weakling is a dancer. Thus,\
"task": "mmlu_flan_cot_fewshot_formal_logic" \ we can eliminate them. Option (B) omits the important detail about Georgia\
\ playing volleyball. Option (D) has all the details presented in the arguments\
\ and is the best English interpretation of the arguments. The answer is (D)."
- question: "Select the best translation into predicate logic: No people drive on Mars.\n\
(A) ~Pd (B) (\u2200x)(Px \u2228 ~Dx) (C) (\u2200x)(Px \u2283 ~Dx) (D) ~Dp"
target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
\ for help. Let\u2019s solve this step by step. Let \u201CP\u201D denote \u201C\
being on Mars\u201D and \u201CD\u201D denote \u201Cdriving on Mars\u201D. Then\
\ let\u2019s consider each option:\nOption (A): ~Pd \u2192 d is not on Mars.\n\
Option (B): (\u2200x)(Px \u2228 ~Dx) \u2192 For all x, x is on Mars and x do\
\ not drive on Mars.\nOption (C): (\u2200x)(Px \u2283 ~Dx) \u2192 For all x,\
\ x is on Mars implies that x do not drive on Mars.\nOption (D): ~Dp: \u2192\
\ p do not drive on Mars.\nOf all these options, Option (C) appears to be the\
\ best and most meaningful interpretation of the argument \u201CNo people drive\
\ on Mars.\u201D The answer is (C).\n\n"
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_formal_logic
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml
View file @ a2af2101
"dataset_name": "global_facts" dataset_name: global_facts
"description": "The following are multiple choice questions (with answers) about global\ description: The following are multiple choice questions (with answers) about global
\ facts.\n\nQ: As of 2017, how many of the world’s 1-year-old children today have\ facts.
\ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\nA: Let's\ fewshot_config:
\ think step by step. We refer to Wikipedia articles on global facts for help. According\ sampler: first_n
\ to data published by the World Health Organization, the nummber of 1-year-old\ samples:
\ children vaccinated in 2017 exceeds 80%. The answer is (A).\n\nQ: As of 2019,\ - question: "As of 2017, how many of the world\u2019s 1-year-old children today have\
\ about what percentage of Americans agree that the state is run for the benefit\ \ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%"
\ of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think step by step.\ target: Let's think step by step. We refer to Wikipedia articles on global facts
\ We refer to Wikipedia articles on global facts for help. In 2019, about 46% percentage\ for help. According to data published by the World Health Organization, the
\ of Americans agree that the state is run for the benefit of all the people. The\ nummber of 1-year-old children vaccinated in 2017 exceeds 80%. The answer is
\ answer is (B).\n\nQ: As of 2019, about what percentage of Russians say it is very\ (A).
\ important to have free media in our country without government/state censorship?\n\ - question: 'As of 2019, about what percentage of Americans agree that the state is
(A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think step by step. We refer to Wikipedia\ run for the benefit of all the people?
\ articles on global facts for help. As of 2019, about 38% of Russians say it is\
\ very important to have free media in our country. The answer is (A).\n\nQ: As\ (A) 31% (B) 46% (C) 61% (D) 76%'
\ of 2015, since 1990 forests have ____ in Europe and have ____ in Africa and the\ target: Let's think step by step. We refer to Wikipedia articles on global facts
\ Americas.\n(A) increased, increased (B) increased, decreased (C) decreased, increased\ for help. In 2019, about 46% percentage of Americans agree that the state is
\ (D) decreased, decreased\nA: Let's think step by step. We refer to Wikipedia articles\ run for the benefit of all the people. The answer is (B).
\ on global facts for help. As of 2015, since 1990 forests have increased in Europe\ - question: 'As of 2019, about what percentage of Russians say it is very important
\ and have decreased in Africa and the Americas. The answer is (B).\n\nQ: Which\ to have free media in our country without government/state censorship?
\ of the following pairs of statements are both true (as of 2019)?\n(A) People tend\
\ to be optimistic about their own future and the future of their nation or the\ (A) 38% (B) 53% (C) 68% (D) 83%'
\ world. (B) People tend to be optimistic about their own future but pessimistic\ target: Let's think step by step. We refer to Wikipedia articles on global facts
\ about the future of their nation or the world. (C) People tend to be pessimistic\ for help. As of 2019, about 38% of Russians say it is very important to have
\ about their own future but optimistic about the future of their nation or the\ free media in our country. The answer is (A).
\ world. (D) People tend to be pessimistic about their own future and the future\ - question: 'As of 2015, since 1990 forests have ____ in Europe and have ____ in Africa
\ of their nation or the world.\nA: Let's think step by step. We refer to Wikipedia\ and the Americas.
\ articles on global facts for help. As of 2019, most people tend to be optimistic\
\ about their own future but pessimistic about the future of their nation or the\ (A) increased, increased (B) increased, decreased (C) decreased, increased (D)
\ world. The answer is (B).\n\n" decreased, decreased'
"group": "mmlu_flan_cot_fewshot_other" target: Let's think step by step. We refer to Wikipedia articles on global facts
"include": "_mmlu_flan_cot_fewshot_template_yaml" for help. As of 2015, since 1990 forests have increased in Europe and have decreased
"task": "mmlu_flan_cot_fewshot_global_facts" in Africa and the Americas. The answer is (B).
- question: 'Which of the following pairs of statements are both true (as of 2019)?
(A) People tend to be optimistic about their own future and the future of their
nation or the world. (B) People tend to be optimistic about their own future
but pessimistic about the future of their nation or the world. (C) People tend
to be pessimistic about their own future but optimistic about the future of
their nation or the world. (D) People tend to be pessimistic about their own
future and the future of their nation or the world.'
target: 'Let''s think step by step. We refer to Wikipedia articles on global facts
for help. As of 2019, most people tend to be optimistic about their own future
but pessimistic about the future of their nation or the world. The answer is
(B).'
tag: mmlu_flan_cot_fewshot_other
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_global_facts
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml
View file @ a2af2101
"dataset_name": "high_school_biology" dataset_name: high_school_biology
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school biology.\n\nQ: In animal cells, which of the following represents the most\ school biology.
\ likely pathway that a secretory protein takes as it is synthesized in a cell?\n\ fewshot_config:
(A) Plasma membrane–Golgi apparatus–ribosome–secretory vesicle–rough ER (B) Ribosome–Golgi\ sampler: first_n
\ apparatus–rough ER–secretory vesicle–plasma membrane (C) Plasma membrane–Golgi\ samples:
\ apparatus–ribosome–secretory vesicle–rough ER (D) Ribosome–rough ER–Golgi apparatus–secretory\ - question: "In animal cells, which of the following represents the most likely pathway\
\ vesicle–plasma membrane\nA: Let's think step by step. Protein synthesis starts\ \ that a secretory protein takes as it is synthesized in a cell?\n(A) Plasma\
\ at the ribosome, so we can eliminate (A) and (C). The ribosome is often in the\ \ membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory vesicle\u2013rough\
\ endoplasmic reticulum and moves from there to the Golgi apparatus, where it is\ \ ER (B) Ribosome\u2013Golgi apparatus\u2013rough ER\u2013secretory vesicle\u2013\
\ modified and packaged into a vesicle. The vesicle then floats to the plasma membrane\ plasma membrane (C) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013\
\ and is secreted. The answer is (D).\n\nQ: A mutation in a bacterial enzyme changed\ secretory vesicle\u2013rough ER (D) Ribosome\u2013rough ER\u2013Golgi apparatus\u2013\
\ a previously polar amino acid into a nonpolar amino acid. This amino acid was\ secretory vesicle\u2013plasma membrane"
\ located at a site distant from the enzyme’s active site. How might this mutation\ target: Let's think step by step. Protein synthesis starts at the ribosome, so
\ alter the enzyme’s substrate specificity?\n(A) By changing the enzyme’s pH optimum\ we can eliminate (A) and (C). The ribosome is often in the endoplasmic reticulum
\ (B) By changing the enzyme’s location in the cell (C) By changing the shape of\ and moves from there to the Golgi apparatus, where it is modified and packaged
\ the protein (D) An amino acid change away from the active site cannot alter the\ into a vesicle. The vesicle then floats to the plasma membrane and is secreted.
\ enzyme’s substrate specificity.\nA: Let's think step by step. A change in an amino\ The answer is (D).
\ acid leads to a change in the primary structure of the protein. A change in the\ - question: "A mutation in a bacterial enzyme changed a previously polar amino acid\
\ primary structure may lead to a change in the secondary and the tertiary structure\ \ into a nonpolar amino acid. This amino acid was located at a site distant\
\ of the protein. A change in the tertiary structure means a change in the shape\ \ from the enzyme\u2019s active site. How might this mutation alter the enzyme\u2019\
\ of the protein, so (C) has to be correct. Since the change does not affect the\ s substrate specificity?\n(A) By changing the enzyme\u2019s pH optimum (B) By\
\ active site of the enzyme, we do not expect the activity of the enzyme to be affected.\ \ changing the enzyme\u2019s location in the cell (C) By changing the shape\
\ The answer is (C).\n\nQ: Which of the following is not a way to form recombinant\ \ of the protein (D) An amino acid change away from the active site cannot alter\
\ DNA?\n(A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation\n\ \ the enzyme\u2019s substrate specificity."
A: Let's think step by step. The introduction of foreign DNA or RNA into bacteria\ target: Let's think step by step. A change in an amino acid leads to a change
\ or eukaryotic cells is a common technique in molecular biology and scientific\ in the primary structure of the protein. A change in the primary structure may
\ research. There are multiple ways foreign DNA can be introduced into cells including\ lead to a change in the secondary and the tertiary structure of the protein.
\ transformation, transduction, conjugation, and transfection. In contrast, (A)\ A change in the tertiary structure means a change in the shape of the protein,
\ is not a way to form DNA: during translation the ribosomes synthesize proteins\ so (C) has to be correct. Since the change does not affect the active site of
\ from RNA. The answer is (A).\n\nQ: Homologous structures are often cited as evidence\ the enzyme, we do not expect the activity of the enzyme to be affected. The
\ for the process of natural selection. All of the following are examples of homologous\ answer is (C).
\ structures EXCEPT\n(A) the wings of a bird and the wings of a bat (B) the flippers\ - question: 'Which of the following is not a way to form recombinant DNA?
\ of a whale and the arms of a man (C) the pectoral fins of a porpoise and the flippers\
\ of a seal (D) the forelegs of an insect and the forelimbs of a dog\nA: Let's think\ (A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation'
\ step by step. ​​Homologous structures are similar physical features in organisms\ target: 'Let''s think step by step. The introduction of foreign DNA or RNA into
\ that share a common ancestor ​​but different functions. Comparisons (B) and (C)\ bacteria or eukaryotic cells is a common technique in molecular biology and
\ are clearly homologous because they share a common ancestor and the structures\ scientific research. There are multiple ways foreign DNA can be introduced into
\ serve different purposes. Bat wings and birg wings are also homologous, while\ cells including transformation, transduction, conjugation, and transfection.
\ they are both wings, the forelimbs serve different purposes. Insects and dogs\ In contrast, (A) is not a way to form DNA: during translation the ribosomes
\ are very far ancestors since one is vertebrate while the other is invertebrate\ synthesize proteins from RNA. The answer is (A).'
\ and the forelimbs serve the same purpose, so they are not homologous. The answer\ - question: 'Homologous structures are often cited as evidence for the process of natural
\ is (D).\n\nQ: Which of the following is not known to be involved in the control\ selection. All of the following are examples of homologous structures EXCEPT
\ of cell division?\n(A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast\
\ cells\nA: Let's think step by step. Normal cells move through the cell cycle in\ (A) the wings of a bird and the wings of a bat (B) the flippers of a whale and
\ a regulated way. At the checkpoint stage, they use information about their own\ the arms of a man (C) the pectoral fins of a porpoise and the flippers of a
\ internal state and cues from the environment around them to decide whether to\ seal (D) the forelegs of an insect and the forelimbs of a dog'
\ proceed with cell division. Cues like these act by changing the activity of core\ target: "Let's think step by step. \u200B\u200BHomologous structures are similar\
\ cell cycle regulators inside the cell. The most common regulators are cyclins\ \ physical features in organisms that share a common ancestor \u200B\u200Bbut\
\ and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division.\ \ different functions. Comparisons (B) and (C) are clearly homologous because\
\ The answer is (D).\n\n" \ they share a common ancestor and the structures serve different purposes.\
"group": "mmlu_flan_cot_fewshot_stem" \ Bat wings and birg wings are also homologous, while they are both wings, the\
"include": "_mmlu_flan_cot_fewshot_template_yaml" \ forelimbs serve different purposes. Insects and dogs are very far ancestors\
"task": "mmlu_flan_cot_fewshot_high_school_biology" \ since one is vertebrate while the other is invertebrate and the forelimbs\
\ serve the same purpose, so they are not homologous. The answer is (D)."
- question: 'Which of the following is not known to be involved in the control of cell
division?
(A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast cells'
target: 'Let''s think step by step. Normal cells move through the cell cycle in
a regulated way. At the checkpoint stage, they use information about their own
internal state and cues from the environment around them to decide whether to
proceed with cell division. Cues like these act by changing the activity of
core cell cycle regulators inside the cell. The most common regulators are cyclins
and cyclin-dependent kinases. Fibroblast cells do not play any role in cell
division. The answer is (D).'
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_biology
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml
View file @ a2af2101
"dataset_name": "high_school_chemistry" dataset_name: high_school_chemistry
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school chemistry.\n\nQ: Which of the following is considered an acid anhydride?\n\ school chemistry.
(A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3\nA: Let's think step by step. An acid anhydride\ fewshot_config:
\ is a compound that is derived by removing water from an acid. The chemical formula\ sampler: first_n
\ for water is H2O, which means that we need to determine which of these options,\ samples:
\ when combined with H2O, forms an acid. SO2, or Sulfur dioxide, when combined with\ - question: 'Which of the following is considered an acid anhydride?
\ H2O, makes H2SO4, or sulfuric acid. The answer is (C).\n\nQ: Which of the following\
\ is expected to be a polar molecule?\n(A) PCl4F (B) BF3 (C) CO2 (D) Si(CH3)4\n\ (A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3'
A: Let's think step by step. A polar molecule is one that has a slightly positive\ target: Let's think step by step. An acid anhydride is a compound that is derived
\ charge on one end of the molecule and a slightly negative charge on the other\ by removing water from an acid. The chemical formula for water is H2O, which
\ end. Boron trifluoride (BF3) has Boron as the center atom and three fluorine atoms\ means that we need to determine which of these options, when combined with H2O,
\ attached to it; it is trigonal planar and symmetric, so it is nonpolar. Carbon\ forms an acid. SO2, or Sulfur dioxide, when combined with H2O, makes H2SO4,
\ Dioxide (CO2) has Carbon as the central atom with double bonds to two Oxygen atoms\ or sulfuric acid. The answer is (C).
\ - this is also symmetrical and therefore nonpolar. The same is the case for tetramethyl\ - question: 'Which of the following is expected to be a polar molecule?
\ silane (SI(CH3)4), which is a Silicon atom surrounded by four methyl groups. The\
\ structure of PCL4F is that Phosphorus is the central atom, attached to four chlorines\ (A) PCl4F (B) BF3 (C) CO2 (D) Si(CH3)4'
\ and one fluorine atom. This is asymmetrical, and therefore has a net dipole and\ target: Let's think step by step. A polar molecule is one that has a slightly
\ is expected to be a polar molecule. The answer is (A).\n\nQ: From the solubility\ positive charge on one end of the molecule and a slightly negative charge on
\ rules, which of the following is true?\n(A) All chlorides, bromides, and iodides\ the other end. Boron trifluoride (BF3) has Boron as the center atom and three
\ are soluble (B) All sulfates are soluble (C) All hydroxides are soluble (D) All\ fluorine atoms attached to it; it is trigonal planar and symmetric, so it is
\ ammonium-containing compounds are soluble\nA: Let's think step by step. The chlorides,\ nonpolar. Carbon Dioxide (CO2) has Carbon as the central atom with double bonds
\ bromides, and iodides of lead, silver, and mercury are not soluble in water. This\ to two Oxygen atoms - this is also symmetrical and therefore nonpolar. The same
\ rules out (A). The sulfates of lead, barium, and calcium are not soluble in water,\ is the case for tetramethyl silane (SI(CH3)4), which is a Silicon atom surrounded
\ which rules out (B). The hydroxides of any metal besides sodium, potassium, ammonium,\ by four methyl groups. The structure of PCL4F is that Phosphorus is the central
\ calcium, and barium are insoluble. This rules out (C). Typically ammonium ions\ atom, attached to four chlorines and one fluorine atom. This is asymmetrical,
\ indicate a soluble ionic substance. The answer is (D).\n\nQ: A new compound is\ and therefore has a net dipole and is expected to be a polar molecule. The answer
\ synthesized and found to be a monoprotic acid with a molar mass of 248 g/mol.\ is (A).
\ When 0.0050 mol of this acid are dissolved in 0.500 L of water, the pH is measured\ - question: 'From the solubility rules, which of the following is true?
\ as 3.89. What is the pKa of this acid?\n(A) 3.89 (B) 7.78 (C) 5.78 (D) 2.33\n\
A: Let's think step by step. Recall that $[A] = [H^{+}]$. Here, this is equal to\ (A) All chlorides, bromides, and iodides are soluble (B) All sulfates are soluble
\ $$10^{-3.89}$. Then we have $K_{a} = $\nrac{[H^{+}][A^{-}]}{[HA]} = \nrac{10^{-3.89}\ (C) All hydroxides are soluble (D) All ammonium-containing compounds are soluble'
\ \\cdot 10^{-3.89}}{10^{-2}}. The resulting exponent is $-3.89 + (-3.89) - (-2)\ target: Let's think step by step. The chlorides, bromides, and iodides of lead,
\ = 5.78$, therefore $K_a = 10^{-5.78}$. The $pK_a$ is the negative log of $K_a$,\ silver, and mercury are not soluble in water. This rules out (A). The sulfates
\ which is equal to $5.78$. The answer is (C).\n\nQ: A solution contains 2.00 mole\ of lead, barium, and calcium are not soluble in water, which rules out (B).
\ of acetic acid, CH3COOH, and 1.00 mole of calcium acetate, Ca(CH3COO)2. The solution\ The hydroxides of any metal besides sodium, potassium, ammonium, calcium, and
\ is able to resist the addition of a small amount of strong acid or strong base\ barium are insoluble. This rules out (C). Typically ammonium ions indicate a
\ with only minor changes in the pH of the solution. Larger quantities of strong\ soluble ionic substance. The answer is (D).
\ acid or strong base can cause a significant change in pH. How many moles of nitric\ - question: 'A new compound is synthesized and found to be a monoprotic acid with a
\ acid, HNO3, may be added before the pH begins to change significantly?\n(A) 0.500\ molar mass of 248 g/mol. When 0.0050 mol of this acid are dissolved in 0.500
\ mole (B) 1.00 mole (C) 2.00 mole (D) 3.00 mole\nA: Let's think step by step. We\ L of water, the pH is measured as 3.89. What is the pKa of this acid?
\ would like to compute the buffer capacity of this solution. First we write the\
\ equation for the ionization of the weak acid, in this case of acetic acid. $CH_{3}COOH\ (A) 3.89 (B) 7.78 (C) 5.78 (D) 2.33'
\ (aq) + H_{2}O \nightarrow H_{3}O^{+} + CH3COO^{-}$. The conjugate base is therefore\ target: "Let's think step by step. Recall that $[A] = [H^{+}]$. Here, this is\
\ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\ \ equal to $$10^{-3.89}$. Then we have $K_{a} = $\nrac{[H^{+}][A^{-}]}{[HA]}\
\ base. Therefore the maximum amount of acid that can be added will be equal to\ \ = \nrac{10^{-3.89} \\cdot 10^{-3.89}}{10^{-2}}. The resulting exponent is\
\ the amount of acetate ion, or 2 moles. The answer is (C).\n\n" \ $-3.89 + (-3.89) - (-2) = 5.78$, therefore $K_a = 10^{-5.78}$. The $pK_a$\
"group": "mmlu_flan_cot_fewshot_stem" \ is the negative log of $K_a$, which is equal to $5.78$. The answer is (C)."
"include": "_mmlu_flan_cot_fewshot_template_yaml" - question: 'A solution contains 2.00 mole of acetic acid, CH3COOH, and 1.00 mole of
"task": "mmlu_flan_cot_fewshot_high_school_chemistry" calcium acetate, Ca(CH3COO)2. The solution is able to resist the addition of
a small amount of strong acid or strong base with only minor changes in the
pH of the solution. Larger quantities of strong acid or strong base can cause
a significant change in pH. How many moles of nitric acid, HNO3, may be added
before the pH begins to change significantly?
(A) 0.500 mole (B) 1.00 mole (C) 2.00 mole (D) 3.00 mole'
target: "Let's think step by step. We would like to compute the buffer capacity\
\ of this solution. First we write the equation for the ionization of the weak\
\ acid, in this case of acetic acid. $CH_{3}COOH (aq) + H_{2}O \nightarrow H_{3}O^{+}\
\ + CH3COO^{-}$. The conjugate base is therefore the acetate ion. The added\
\ strong acid, Nitric acid, will react with the conjugate base. Therefore the\
\ maximum amount of acid that can be added will be equal to the amount of acetate\
\ ion, or 2 moles. The answer is (C).\n\n"
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_chemistry
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml
View file @ a2af2101
"dataset_name": "high_school_computer_science" dataset_name: high_school_computer_science
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school computer science.\n\nQ: Which of the following is an example of the use\ school computer science.
\ of a device on the Internet of Things (IoT) ?\n(A) A car alerts a driver that\ fewshot_config:
\ it is about to hit an object. (B) A hiker uses a G P S watch to keep track of\ sampler: first_n
\ her position. (C) A refrigerator orders milk from an online delivery service when\ samples:
\ the milk in the refrigerator is almost gone. (D) A runner uses a watch with optical\ - question: 'Which of the following is an example of the use of a device on the Internet
\ sensors to monitor his heart rate.\nA: Let's think step by step. The term Internet\ of Things (IoT) ?
\ of Things (IoT) refers to common devices which are connected to the internet,\
\ enabling new functionality. Choice A is incorrect because it does not describe\ (A) A car alerts a driver that it is about to hit an object. (B) A hiker uses
\ an internet connected device. In choice B, the watch is only described as having\ a G P S watch to keep track of her position. (C) A refrigerator orders milk
\ GPS functionality but no internet connectivity. Choice C describes a common device\ from an online delivery service when the milk in the refrigerator is almost
\ (a refrigerator) which has internet connectivity enabling new functionality (online\ gone. (D) A runner uses a watch with optical sensors to monitor his heart rate.'
\ ordering). Choice D does not mention internet connectivity for the watch, only\ target: Let's think step by step. The term Internet of Things (IoT) refers to
\ optical sensors. The answer is (C).\n\nQ: Many Web browsers allow users to open\ common devices which are connected to the internet, enabling new functionality.
\ anonymous windows. During a browsing session in an anonymous window, the browser\ Choice A is incorrect because it does not describe an internet connected device.
\ does not record a browsing history or a list of downloaded files. When the anonymous\ In choice B, the watch is only described as having GPS functionality but no
\ window is exited, cookies created during the session are deleted. Which of the\ internet connectivity. Choice C describes a common device (a refrigerator) which
\ following statements about browsing sessions in an anonymous window is true?\n\ has internet connectivity enabling new functionality (online ordering). Choice
(A) The activities of a user browsing in an anonymous window will not be visible\ D does not mention internet connectivity for the watch, only optical sensors.
\ to people who monitor the user's network, such as the system administrator. (B)\ The answer is (C).
\ Items placed in a Web store's shopping cart for future purchase during the anonymous\ - question: 'Many Web browsers allow users to open anonymous windows. During a browsing
\ browsing session will not be saved on the user's computer. (C) A user will not\ session in an anonymous window, the browser does not record a browsing history
\ be able to log in to e-mail or social media accounts during the anonymous browsing\ or a list of downloaded files. When the anonymous window is exited, cookies
\ session. (D) A user browsing in an anonymous window will be protected from viruses\ created during the session are deleted. Which of the following statements about
\ launched from any web sites visited or files downloaded.\nA: Let's think step\ browsing sessions in an anonymous window is true?
\ by step. Choice A is incorrect as it only describes network traffic, which an\
\ anonymous browser does not change. Choice B is correct as it correctly describes\ (A) The activities of a user browsing in an anonymous window will not be visible
\ how an anonymous browser will prevent saving data on the user’s computer after\ to people who monitor the user''s network, such as the system administrator.
\ the session is ended. Choice C is incorrect because an anonymous browser will\ (B) Items placed in a Web store''s shopping cart for future purchase during
\ not prevent logging in to email or social media accounts. Choice D is incorrect\ the anonymous browsing session will not be saved on the user''s computer. (C)
\ because an anonymous browser in itself performs no virus protection. The answer\ A user will not be able to log in to e-mail or social media accounts during
\ is (B).\n\nQ: In the program below, the initial value of X is 5 and the initial\ the anonymous browsing session. (D) A user browsing in an anonymous window will
\ value of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\ be protected from viruses launched from any web sites visited or files downloaded.'
\ DISPLAY (\"Hotel\")\n } ELSE {\n IF (Y > 0){\n DISPLAY (\"November\")\n }\ target: "Let's think step by step. Choice A is incorrect as it only describes\
\ ELSE {\n DISPLAY (\"Yankee\")\n }\n }\n}\nWhat is displayed as a result of\ \ network traffic, which an anonymous browser does not change. Choice B is correct\
\ running the program?\n(A) Foxtrot (B) Hotel (C) November (D) Yankee\nA: Let's\ \ as it correctly describes how an anonymous browser will prevent saving data\
\ think step by step. Because X has the value 5, the first conditional IF (X < 0)\ \ on the user\u2019s computer after the session is ended. Choice C is incorrect\
\ is false, so we move to the first ELSE clause. Because X is 5 and Y is 10, the\ \ because an anonymous browser will not prevent logging in to email or social\
\ second conditional IF (X > Y) is false, so we move to the following ELSE clause.\ \ media accounts. Choice D is incorrect because an anonymous browser in itself\
\ Since Y is 10, the conditional IF (Y > 0) is true, so the command DISPLAY (\"\ \ performs no virus protection. The answer is (B)."
November\") is executed. The answer is (C).\n\nQ: What is the output of \"abc\"\ - question: "In the program below, the initial value of X is 5 and the initial value\
[::-1] in Python 3?\n(A) Error (B) abc (C) cba (D) c\nA: Let's think step by step.\ \ of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\
\ We know that the slicing operator [::-1] takes all of the elements in the string\ \ DISPLAY (\"Hotel\")\n } ELSE {\n IF (Y > 0){\n DISPLAY (\"November\")\n\
\ in reverse order, so we reverse the order of the string \"abc\", resulting in\ \ } ELSE {\n DISPLAY (\"Yankee\")\n }\n }\n}\nWhat is displayed as a result\
\ \"cba\". The answer is (C).\n\nQ: A list of numbers has n elements, indexed from\ \ of running the program?\n(A) Foxtrot (B) Hotel (C) November (D) Yankee"
\ 1 to n. The following algorithm is intended to display the number of elements\ target: Let's think step by step. Because X has the value 5, the first conditional
\ in the list that have a value greater than 100. The algorithm uses the variables\ IF (X < 0) is false, so we move to the first ELSE clause. Because X is 5 and
\ count and position. Steps 3 and 4 are missing.\n Step 1: Set count to 0 and position\ Y is 10, the second conditional IF (X > Y) is false, so we move to the following
\ to 1.\n Step 2: If the value of the element at index position is greater than\ ELSE clause. Since Y is 10, the conditional IF (Y > 0) is true, so the command
\ 100, increase the value of count by 1.\n Step 3: (missing step)\n Step 4: (missing\ DISPLAY ("November") is executed. The answer is (C).
\ step)\n Step 5: Display the value of count.\nWhich of the following could be used\ - question: 'What is the output of "abc"[::-1] in Python 3?
\ to replace steps 3 and 4 so that the algorithm works as intended?\n(A) Step 3:\
\ Increase the value of position by 1.\n Step 4: Repeat steps 2 and 3 until the\ (A) Error (B) abc (C) cba (D) c'
\ value of count is greater than 100.\n(B) Step 3: Increase the value of position\ target: Let's think step by step. We know that the slicing operator [::-1] takes
\ by 1.\n Step 4: Repeat steps 2 and 3 until the value of position is greater than\ all of the elements in the string in reverse order, so we reverse the order
\ n.\n(C) Step 3: Repeat step 2 until the value of count is greater than 100.\n\ of the string "abc", resulting in "cba". The answer is (C).
\ Step 4: Increase the value of position by 1.\n(D) Step 3: Repeat step 2 until\ - question: "A list of numbers has n elements, indexed from 1 to n. The following algorithm\
\ the value of position is greater than n.\n Step 4: Increase the value of count\ \ is intended to display the number of elements in the list that have a value\
\ by 1.\nA: Let's think step by step. Choice A is incorrect, because its Step 4\ \ greater than 100. The algorithm uses the variables count and position. Steps\
\ has an incorrect termination condition, stopping when count is greater than 100.\ \ 3 and 4 are missing.\n Step 1: Set count to 0 and position to 1.\n Step 2:\
\ We need to stop after inspecting all elements in the list. Choice B is correct\ \ If the value of the element at index position is greater than 100, increase\
\ because it correctly increments both count and position, and correctly repeats\ \ the value of count by 1.\n Step 3: (missing step)\n Step 4: (missing step)\n\
\ these steps and terminates when all elements in the list have been inspected.\ \ Step 5: Display the value of count.\nWhich of the following could be used\
\ Choice C is incorrect because it incorrectly increments the variable count until\ \ to replace steps 3 and 4 so that the algorithm works as intended?\n(A) Step\
\ its value is greater than 100, regardless of the elements in the list. Choice\ \ 3: Increase the value of position by 1.\n Step 4: Repeat steps 2 and 3 until\
\ D is incorrect because its step 3 does not increment the value of position, so\ \ the value of count is greater than 100.\n(B) Step 3: Increase the value of\
\ it will repeat forever. The answer is (B).\n\n" \ position by 1.\n Step 4: Repeat steps 2 and 3 until the value of position\
"group": "mmlu_flan_cot_fewshot_stem" \ is greater than n.\n(C) Step 3: Repeat step 2 until the value of count is\
"include": "_mmlu_flan_cot_fewshot_template_yaml" \ greater than 100.\n Step 4: Increase the value of position by 1.\n(D) Step\
"task": "mmlu_flan_cot_fewshot_high_school_computer_science" \ 3: Repeat step 2 until the value of position is greater than n.\n Step 4:\
\ Increase the value of count by 1."
target: 'Let''s think step by step. Choice A is incorrect, because its Step 4
has an incorrect termination condition, stopping when count is greater than
100. We need to stop after inspecting all elements in the list. Choice B is
correct because it correctly increments both count and position, and correctly
repeats these steps and terminates when all elements in the list have been inspected.
Choice C is incorrect because it incorrectly increments the variable count until
its value is greater than 100, regardless of the elements in the list. Choice
D is incorrect because its step 3 does not increment the value of position,
so it will repeat forever. The answer is (B).'
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_computer_science
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml
View file @ a2af2101
"dataset_name": "high_school_european_history" dataset_name: high_school_european_history
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school european history.\n\nQ: This question refers to the following information.\n\ school european history.
Albeit the king's Majesty justly and rightfully is and ought to be the supreme head\ fewshot_config:
\ of the Church of England, and so is recognized by the clergy of this realm in\ sampler: first_n
\ their convocations, yet nevertheless, for corroboration and confirmation thereof,\ samples:
\ and for increase of virtue in Christ's religion within this realm of England,\ - question: 'This question refers to the following information.
\ and to repress and extirpate all errors, heresies, and other enormities and abuses\
\ heretofore used in the same, be it enacted, by authority of this present Parliament,\ Albeit the king''s Majesty justly and rightfully is and ought to be the supreme
\ that the king, our sovereign lord, his heirs and successors, kings of this realm,\ head of the Church of England, and so is recognized by the clergy of this realm
\ shall be taken, accepted, and reputed the only supreme head in earth of the Church\ in their convocations, yet nevertheless, for corroboration and confirmation
\ of England, called Anglicans Ecclesia; and shall have and enjoy, annexed and united\ thereof, and for increase of virtue in Christ''s religion within this realm
\ to the imperial crown of this realm, as well the title and style thereof, as all\ of England, and to repress and extirpate all errors, heresies, and other enormities
\ honors, dignities, preeminences, jurisdictions, privileges, authorities, immunities,\ and abuses heretofore used in the same, be it enacted, by authority of this
\ profits, and commodities to the said dignity of the supreme head of the same Church\ present Parliament, that the king, our sovereign lord, his heirs and successors,
\ belonging and appertaining; and that our said sovereign lord, his heirs and successors,\ kings of this realm, shall be taken, accepted, and reputed the only supreme
\ kings of this realm, shall have full power and authority from time to time to\ head in earth of the Church of England, called Anglicans Ecclesia; and shall
\ visit, repress, redress, record, order, correct, restrain, and amend all such\ have and enjoy, annexed and united to the imperial crown of this realm, as well
\ errors, heresies, abuses, offenses, contempts, and enormities, whatsoever they\ the title and style thereof, as all honors, dignities, preeminences, jurisdictions,
\ be, which by any manner of spiritual authority or jurisdiction ought or may lawfully\ privileges, authorities, immunities, profits, and commodities to the said dignity
\ be reformed, repressed, ordered, redressed, corrected, restrained, or amended,\ of the supreme head of the same Church belonging and appertaining; and that
\ most to the pleasure of Almighty God, the increase of virtue in Christ's religion,\ our said sovereign lord, his heirs and successors, kings of this realm, shall
\ and for the conservation of the peace, unity, and tranquility of this realm; any\ have full power and authority from time to time to visit, repress, redress,
\ usage, foreign land, foreign authority, prescription, or any other thing or things\ record, order, correct, restrain, and amend all such errors, heresies, abuses,
\ to the contrary hereof notwithstanding.\nEnglish Parliament, Act of Supremacy,\ offenses, contempts, and enormities, whatsoever they be, which by any manner
\ 1534\nFrom the passage, one may infer that the English Parliament wished to argue\ of spiritual authority or jurisdiction ought or may lawfully be reformed, repressed,
\ that the Act of Supremacy would\n(A) give the English king a new position of authority\ ordered, redressed, corrected, restrained, or amended, most to the pleasure
\ (B) give the position of head of the Church of England to Henry VIII alone and\ of Almighty God, the increase of virtue in Christ''s religion, and for the conservation
\ exclude his heirs (C) establish Calvinism as the one true theology in England\ of the peace, unity, and tranquility of this realm; any usage, foreign land,
\ (D) end various forms of corruption plaguing the Church in England\nA: Let's think\ foreign authority, prescription, or any other thing or things to the contrary
\ step by step. We refer to Wikipedia articles on european history for help. The\ hereof notwithstanding.
\ Act of Supremacy states that it grants authority to the king \"to repress and\
\ extirpate all errors, heresies, and other enormities and abuses\", referring to\ English Parliament, Act of Supremacy, 1534
\ the corruption in the Church of England. The answer is (D).\n\nQ: This question\
\ refers to the following information.\nRead the following excerpt.\nThe revolutionary\ From the passage, one may infer that the English Parliament wished to argue
\ seed had penetrated into every country and spread more or less. It was greatly\ that the Act of Supremacy would
\ developed under the régime of the military despotism of Bonaparte. His conquests\
\ displaced a number of laws, institutions, and customs; broke through bonds sacred\ (A) give the English king a new position of authority (B) give the position
\ among all nations, strong enough to resist time itself; which is more than can\ of head of the Church of England to Henry VIII alone and exclude his heirs (C)
\ be said of certain benefits conferred by these innovators.\nThe monarchs will\ establish Calvinism as the one true theology in England (D) end various forms
\ fulfil the duties imposed upon them by Him who, by entrusting them with power,\ of corruption plaguing the Church in England'
\ has charged them to watch over the maintenance of justice, and the rights of all,\ target: Let's think step by step. We refer to Wikipedia articles on european history
\ to avoid the paths of error, and tread firmly in the way of truth. Placed beyond\ for help. The Act of Supremacy states that it grants authority to the king "to
\ the passions which agitate society, it is in days of trial chiefly that they are\ repress and extirpate all errors, heresies, and other enormities and abuses",
\ called upon to despoil realities of their false appearances, and to show themselves\ referring to the corruption in the Church of England. The answer is (D).
\ as they are, fathers invested with the authority belonging by right to the heads\ - question: "This question refers to the following information.\nRead the following\
\ of families, to prove that, in days of mourning, they know how to be just, wise,\ \ excerpt.\nThe revolutionary seed had penetrated into every country and spread\
\ and therefore strong, and that they will not abandon the people whom they ought\ \ more or less. It was greatly developed under the r\xE9gime of the military\
\ to govern to be the sport of factions, to error and its consequences, which must\ \ despotism of Bonaparte. His conquests displaced a number of laws, institutions,\
\ involve the loss of society.\nUnion between the monarchs is the basis of the policy\ \ and customs; broke through bonds sacred among all nations, strong enough to\
\ which must now be followed to save society from total ruin. . . .\nLet them not\ \ resist time itself; which is more than can be said of certain benefits conferred\
\ confound concessions made to parties with the good they ought to do for their\ \ by these innovators.\nThe monarchs will fulfil the duties imposed upon them\
\ people, in modifying, according to their recognized needs, such branches of the\ \ by Him who, by entrusting them with power, has charged them to watch over\
\ administration as require it.\nLet them be just, but strong; beneficent, but strict.\n\ \ the maintenance of justice, and the rights of all, to avoid the paths of error,\
Let them maintain religious principles in all their purity, and not allow the faith\ \ and tread firmly in the way of truth. Placed beyond the passions which agitate\
\ to be attacked and morality interpreted according to the social contract or the\ \ society, it is in days of trial chiefly that they are called upon to despoil\
\ visions of foolish sectarians.\nLet them suppress Secret Societies; that gangrene\ \ realities of their false appearances, and to show themselves as they are,\
\ of society.\n—Klemens von Metternich, Political Confession of Faith, 1820\nWhich\ \ fathers invested with the authority belonging by right to the heads of families,\
\ of the following was the greatest cause of the fears expressed by Metternich in\ \ to prove that, in days of mourning, they know how to be just, wise, and therefore\
\ the document above?\n(A) The ideas of personal liberty and nationalism conceived\ \ strong, and that they will not abandon the people whom they ought to govern\
\ during the Enlightenment resulted in radical revolutions that could spread throughout\ \ to be the sport of factions, to error and its consequences, which must involve\
\ Europe. (B) The conquest of Europe by Napoleon led to the creation of new factions\ \ the loss of society.\nUnion between the monarchs is the basis of the policy\
\ and shifted the European balance of power. (C) The power of monarchs had grown\ \ which must now be followed to save society from total ruin. . . .\nLet them\
\ to the point where it needed to be checked by other powers within each nation\ \ not confound concessions made to parties with the good they ought to do for\
\ or domination of civilians would occur. (D) The rising and falling economic cycle\ \ their people, in modifying, according to their recognized needs, such branches\
\ of the newly emerging capitalist economy could lead to civilian unrest that must\ \ of the administration as require it.\nLet them be just, but strong; beneficent,\
\ be suppressed.\nA: Let's think step by step. We refer to Wikipedia articles on\ \ but strict.\nLet them maintain religious principles in all their purity, and\
\ european history for help. The fears of revolution in early 19th century Europe\ \ not allow the faith to be attacked and morality interpreted according to the\
\ expressed by Klemens von Metternich, a conservative Austrian statesman, were a\ \ social contract or the visions of foolish sectarians.\nLet them suppress Secret\
\ direct result of the age of Enlightenment, a period of European history where\ \ Societies; that gangrene of society.\n\u2014Klemens von Metternich, Political\
\ the absolute power of the monarchy was challenged with ideas of individual liberty\ \ Confession of Faith, 1820\nWhich of the following was the greatest cause of\
\ and nationalism, leading to the French revolution and its effects all over Europe.\ \ the fears expressed by Metternich in the document above?\n(A) The ideas of\
\ The answer is (A).\n\nQ: This question refers to the following information.\n\ \ personal liberty and nationalism conceived during the Enlightenment resulted\
The excerpts below are from the Navigation Acts of 1651.\n[A]fter the first day\ \ in radical revolutions that could spread throughout Europe. (B) The conquest\
\ of December, one thousand six hundred fifty and one, and from thence forwards,\ \ of Europe by Napoleon led to the creation of new factions and shifted the\
\ no goods or commodities whatsoever of the growth, production or manufacture of\ \ European balance of power. (C) The power of monarchs had grown to the point\
\ Asia, Africa or America, or of any part thereof; or of any islands belonging to\ \ where it needed to be checked by other powers within each nation or domination\
\ them, or which are described or laid down in the usual maps or cards of those\ \ of civilians would occur. (D) The rising and falling economic cycle of the\
\ places, as well of the English plantations as others, shall be imported or brought\ \ newly emerging capitalist economy could lead to civilian unrest that must\
\ into this Commonwealth of England, or into Ireland, or any other lands, islands,\ \ be suppressed."
\ plantations, or territories to this Commonwealth belonging, or in their possession,\ target: Let's think step by step. We refer to Wikipedia articles on european history
\ in any other ship or ships, vessel or vessels whatsoever, but only in such as\ for help. The fears of revolution in early 19th century Europe expressed by
\ do truly and without fraud belong only to the people of this Commonwealth, or\ Klemens von Metternich, a conservative Austrian statesman, were a direct result
\ the plantations thereof, as the proprietors or right owners thereof; and whereof\ of the age of Enlightenment, a period of European history where the absolute
\ the master and mariners are also of the people of this Commonwealth, under the\ power of the monarchy was challenged with ideas of individual liberty and nationalism,
\ penalty of the forfeiture and loss of all the goods that shall be imported contrary\ leading to the French revolution and its effects all over Europe. The answer
\ to this act, , , ,\n[N]o goods or commodities of the growth, production, or manufacture\ is (A).
\ of Europe, or of any part thereof, shall after the first day of December, one\ - question: 'This question refers to the following information.
\ thousand six hundred fifty and one, be imported or brought into this Commonwealth\
\ of England, or any other lands or territories to this Commonwealth belonging,\ The excerpts below are from the Navigation Acts of 1651.
\ or in their possession, in any ship or ships, vessel or vessels whatsoever, but\
\ in such as do truly and without fraud belong only to the people of this Commonwealth,\ [A]fter the first day of December, one thousand six hundred fifty and one, and
\ and in no other, except only such foreign ships and vessels as do truly and properly\ from thence forwards, no goods or commodities whatsoever of the growth, production
\ belong to the people of that country or place, of which the said goods are the\ or manufacture of Asia, Africa or America, or of any part thereof; or of any
\ growth, production or manufacture.\nWhich of the following best describes the\ islands belonging to them, or which are described or laid down in the usual
\ outcome of the Navigation Acts of 1651?\n(A) They served as a catalyst for the\ maps or cards of those places, as well of the English plantations as others,
\ growth of English shipping and overseas trade, but did little to limit the prospects\ shall be imported or brought into this Commonwealth of England, or into Ireland,
\ of the Dutch in the seventeenth century. (B) They brought about almost immediate\ or any other lands, islands, plantations, or territories to this Commonwealth
\ hardships for the Dutch economy as their dominance of overseas trade quickly ended.\ belonging, or in their possession, in any other ship or ships, vessel or vessels
\ (C) They were rescinded during the restoration of the Stuarts as they sought normal\ whatsoever, but only in such as do truly and without fraud belong only to the
\ diplomatic relations with the Dutch so not as to need Parliament's financial support\ people of this Commonwealth, or the plantations thereof, as the proprietors
\ for war. (D) They led to nearly a century of recurrent war between England and\ or right owners thereof; and whereof the master and mariners are also of the
\ the Netherlands, which would not end until after American independence.\nA: Let's\ people of this Commonwealth, under the penalty of the forfeiture and loss of
\ think step by step. We refer to Wikipedia articles on european history for help.\ all the goods that shall be imported contrary to this act, , , ,
\ The Navigation Acts of 1651 helped English shipping by restricting the ability\
\ of ships from other European countries, especially the Dutch, to transport goods\ [N]o goods or commodities of the growth, production, or manufacture of Europe,
\ from colonies in Asia and Africa into England. The answer is (A).\n\nQ: This question\ or of any part thereof, shall after the first day of December, one thousand
\ refers to the following information.\nIn Russia there was nothing going on well,\ six hundred fifty and one, be imported or brought into this Commonwealth of
\ and [Souvarine] was in despair over the news he had received. His old companions\ England, or any other lands or territories to this Commonwealth belonging, or
\ were all turning to the politicians; the famous Nihilists who made Europe tremble-sons\ in their possession, in any ship or ships, vessel or vessels whatsoever, but
\ of village priests, of the lower middle class, of tradesmen-could not rise above\ in such as do truly and without fraud belong only to the people of this Commonwealth,
\ the idea of national liberation, and seemed to believe that the world would be\ and in no other, except only such foreign ships and vessels as do truly and
\ delivered-when they had killed their despot&…\n\"Foolery! They'll never get out\ properly belong to the people of that country or place, of which the said goods
\ of it with their foolery.\"\nThen, lowering his voice still more, in a few bitter\ are the growth, production or manufacture.
\ words he described his old dream of fraternity. He had renounced his rank and\
\ his fortune; he had gone among workmen, only in the hope of seeing at last the\ Which of the following best describes the outcome of the Navigation Acts of
\ foundation of a new society of labour in common. All the sous in his pockets had\ 1651?
\ long gone to the urchins of the settlement; he had been as tender as a brother\
\ with the colliers, smiling at their suspicion, winning them over by his quiet\ (A) They served as a catalyst for the growth of English shipping and overseas
\ workmanlike ways and his dislike of chattering. But decidedly the fusion had not\ trade, but did little to limit the prospects of the Dutch in the seventeenth
\ taken place.\nHis voice changed, his eyes grew bright, he fixed them on étienne,\ century. (B) They brought about almost immediate hardships for the Dutch economy
\ directly addressing him:\n\"Now, do you understand that? These hatworkers at Marseilles\ as their dominance of overseas trade quickly ended. (C) They were rescinded
\ who have won the great lottery prize of a hundred thousand francs have gone off\ during the restoration of the Stuarts as they sought normal diplomatic relations
\ at once and invested it, declaring that they are going to live without doing anything!\ with the Dutch so not as to need Parliament''s financial support for war. (D)
\ Yes, that is your idea, all of you French workmen; you want to unearth a treasure\ They led to nearly a century of recurrent war between England and the Netherlands,
\ in order to devour it alone afterwards in some lazy, selfish corner. You may cry\ which would not end until after American independence.'
\ out as much as you like against the rich, you haven't got courage enough to give\ target: Let's think step by step. We refer to Wikipedia articles on european history
\ back to the poor the money that luck brings you. You will never be worthy of happiness\ for help. The Navigation Acts of 1651 helped English shipping by restricting
\ as long as you own anything, and your hatred of the bourgeois proceeds solely\ the ability of ships from other European countries, especially the Dutch, to
\ from an angry desire to be bourgeois yourselves in their place.\"\némile Zola,\ transport goods from colonies in Asia and Africa into England. The answer is
\ French writer, Germinal, 1885\nThe passage displays the direct concern for the\ (A).
\ welfare of the working classes that was typically a part of which movement?\n\ - question: "This question refers to the following information.\nIn Russia there was\
(A) Capitalist (B) Scientific (C) Communist (D) Existentialist\nA: Let's think step\ \ nothing going on well, and [Souvarine] was in despair over the news he had\
\ by step. We refer to Wikipedia articles on european history for help. The modern\ \ received. His old companions were all turning to the politicians; the famous\
\ Communist movement aims to establish a classless society based on communal ownership\ \ Nihilists who made Europe tremble-sons of village priests, of the lower middle\
\ and distribution of property and means of production, thereby especially benefiting\ \ class, of tradesmen-could not rise above the idea of national liberation,\
\ the working classes. The answer is (C).\n\nQ: This question refers to the following\ \ and seemed to believe that the world would be delivered-when they had killed\
\ information.\nThe following excerpt is from a pamphlet.\nYou will do me the justice\ \ their despot&\u2026\n\"Foolery! They'll never get out of it with their foolery.\"\
\ to remember, that I have always strenuously supported the Right of every man to\ \nThen, lowering his voice still more, in a few bitter words he described his\
\ his own opinion, however different that opinion might be to mine. He who denies\ \ old dream of fraternity. He had renounced his rank and his fortune; he had\
\ to another this right, makes a slave of himself to his present opinion, because\ \ gone among workmen, only in the hope of seeing at last the foundation of a\
\ he precludes himself the right of changing it.\nThe most formidable weapon against\ \ new society of labour in common. All the sous in his pockets had long gone\
\ errors of every kind is Reason. I have never used any other, and I trust I never\ \ to the urchins of the settlement; he had been as tender as a brother with\
\ shall.\nThe circumstance that has now taken place in France of the total abolition\ \ the colliers, smiling at their suspicion, winning them over by his quiet workmanlike\
\ of the whole national order of priesthood, and of everything appertaining to compulsive\ \ ways and his dislike of chattering. But decidedly the fusion had not taken\
\ systems of religion, and compulsive articles of faith, has not only precipitated\ \ place.\nHis voice changed, his eyes grew bright, he fixed them on \xE9tienne,\
\ my intention, but rendered a work of this kind exceedingly necessary, lest in\ \ directly addressing him:\n\"Now, do you understand that? These hatworkers\
\ the general wreck of superstition, of false systems of government, and false theology,\ \ at Marseilles who have won the great lottery prize of a hundred thousand francs\
\ we lose sight of morality, of humanity, and of the theology that is true.\nI believe\ \ have gone off at once and invested it, declaring that they are going to live\
\ in one God, and no more; and I hope for happiness beyond this life.\nI believe\ \ without doing anything! Yes, that is your idea, all of you French workmen;\
\ in the equality of man; and I believe that religious duties consist in doing justice,\ \ you want to unearth a treasure in order to devour it alone afterwards in some\
\ loving mercy, and endeavoring to make our fellow-creatures happy.\nI do not believe\ \ lazy, selfish corner. You may cry out as much as you like against the rich,\
\ in the creed professed by the Jewish church, by the Roman church, by the Greek\ \ you haven't got courage enough to give back to the poor the money that luck\
\ church, by the Turkish church, by the Protestant church, nor by any church that\ \ brings you. You will never be worthy of happiness as long as you own anything,\
\ I know of. My own mind is my own church.\nAll national institutions of churches,\ \ and your hatred of the bourgeois proceeds solely from an angry desire to be\
\ whether Jewish, Christian or Turkish, appear to me no other than human inventions,\ \ bourgeois yourselves in their place.\"\n\xE9mile Zola, French writer, Germinal,\
\ set up to terrify and enslave mankind, and monopolize power and profit.\nI do\ \ 1885\nThe passage displays the direct concern for the welfare of the working\
\ not mean by this declaration to condemn those who believe otherwise; they have\ \ classes that was typically a part of which movement?\n(A) Capitalist (B) Scientific\
\ the same right to their belief as I have to mine.\n—Thomas Paine, The Age of Reason,\ \ (C) Communist (D) Existentialist"
\ 1794–1795\nWhich of the following Enlightenment philosophes designed a system\ target: Let's think step by step. We refer to Wikipedia articles on european history
\ of checks and balances for government to avoid abuses of power?\n(A) Jean Jacques\ for help. The modern Communist movement aims to establish a classless society
\ Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\nA: Let's\ based on communal ownership and distribution of property and means of production,
\ think step by step. We refer to Wikipedia articles on european history for help.\ thereby especially benefiting the working classes. The answer is (C).
\ Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\ - question: "This question refers to the following information.\nThe following excerpt\
\ against the monoplization of power and advocated for a system of checks and balances\ \ is from a pamphlet.\nYou will do me the justice to remember, that I have always\
\ in government to prevent the rise of despotism. The answer is (B).\n\n" \ strenuously supported the Right of every man to his own opinion, however different\
"group": "mmlu_flan_cot_fewshot_humanities" \ that opinion might be to mine. He who denies to another this right, makes\
"include": "_mmlu_flan_cot_fewshot_template_yaml" \ a slave of himself to his present opinion, because he precludes himself the\
"task": "mmlu_flan_cot_fewshot_high_school_european_history" \ right of changing it.\nThe most formidable weapon against errors of every\
\ kind is Reason. I have never used any other, and I trust I never shall.\n\
The circumstance that has now taken place in France of the total abolition of\
\ the whole national order of priesthood, and of everything appertaining to\
\ compulsive systems of religion, and compulsive articles of faith, has not\
\ only precipitated my intention, but rendered a work of this kind exceedingly\
\ necessary, lest in the general wreck of superstition, of false systems of\
\ government, and false theology, we lose sight of morality, of humanity, and\
\ of the theology that is true.\nI believe in one God, and no more; and I hope\
\ for happiness beyond this life.\nI believe in the equality of man; and I believe\
\ that religious duties consist in doing justice, loving mercy, and endeavoring\
\ to make our fellow-creatures happy.\nI do not believe in the creed professed\
\ by the Jewish church, by the Roman church, by the Greek church, by the Turkish\
\ church, by the Protestant church, nor by any church that I know of. My own\
\ mind is my own church.\nAll national institutions of churches, whether Jewish,\
\ Christian or Turkish, appear to me no other than human inventions, set up\
\ to terrify and enslave mankind, and monopolize power and profit.\nI do not\
\ mean by this declaration to condemn those who believe otherwise; they have\
\ the same right to their belief as I have to mine.\n\u2014Thomas Paine, The\
\ Age of Reason, 1794\u20131795\nWhich of the following Enlightenment philosophes\
\ designed a system of checks and balances for government to avoid abuses of\
\ power?\n(A) Jean Jacques Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft\
\ (D) Adam Smith"
target: 'Let''s think step by step. We refer to Wikipedia articles on european
history for help. Baron Montesquieu was a 18th centrury French philsopher who
wrote extensively against the monoplization of power and advocated for a system
of checks and balances in government to prevent the rise of despotism. The answer
is (B).'
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_european_history
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml
View file @ a2af2101
"dataset_name": "high_school_geography" dataset_name: high_school_geography
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school geography.\n\nQ: Which one of the following items is an example of nonmaterial\ school geography.
\ culture?\n(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).\n\ fewshot_config:
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\ sampler: first_n
\ Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not\ samples:
\ physical objects. The answer is (C).\n\nQ: During the third stage of the demographic\ - question: 'Which one of the following items is an example of nonmaterial culture?
\ transition model, which of the following is true?\n(A) Birth rates increase and\
\ population growth rate is less rapid. (B) Birth rates decline and population growth\ (A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).'
\ rate is less rapid. (C) Birth rates increase and population growth rate increases.\ target: Let's think step by step. We refer to Wikipedia articles on geography
\ (D) Birth rates decrease and population growth rate increases.\nA: Let's think\ for help. Nonmaterial culture consists of cultural ideas, beliefs or symbols
\ step by step. We refer to Wikipedia articles on geography for help. The demographic\ that are not physical objects. The answer is (C).
\ transition model models the five different stages of population growth as a country\ - question: 'During the third stage of the demographic transition model, which of the
\ goes through economic development, where the third stage refers to a period of\ following is true?
\ declining birth rates and lower population growth. The answer is (B).\n\nQ: The\
\ practice of hiring a foreign third-party service provider to run an operation\ (A) Birth rates increase and population growth rate is less rapid. (B) Birth
\ is called\n(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.\n\ rates decline and population growth rate is less rapid. (C) Birth rates increase
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\ and population growth rate increases. (D) Birth rates decrease and population
\ \"Offshoring\" literally means to move or base some of the activities or processes\ growth rate increases.'
\ of a company to a foreign country. The answer is (B).\n\nQ: Which of the following\ target: Let's think step by step. We refer to Wikipedia articles on geography
\ statements is NOT accurate regarding the services provided by local governments\ for help. The demographic transition model models the five different stages
\ in the United States?\n(A) Duplication of efforts occurs often. (B) Social problems\ of population growth as a country goes through economic development, where the
\ of the central city spill over into the surrounding residential suburbs. (C) Inefficiency\ third stage refers to a period of declining birth rates and lower population
\ in providing services occurs often. (D) One neighborhood's efforts to reduce pollution\ growth. The answer is (B).
\ are always supported by neighboring communities.\nA: Let's think step by step.\ - question: 'The practice of hiring a foreign third-party service provider to run an
\ We refer to Wikipedia articles on geography for help. There may be economic, social\ operation is called
\ or political reasons for two neighboring communities and their local governments\
\ not agreeing to pollution reduction efforts initiated by one of them. The answer\ (A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.'
\ is (D).\n\nQ: The rate of natural increase of a population is found by subtracting\ target: Let's think step by step. We refer to Wikipedia articles on geography
\ the\n(A) crude death rate from the crude birth date. (B) crude birth rate from\ for help. "Offshoring" literally means to move or base some of the activities
\ the crude death rate. (C) doubling time from the crude birth rate. (D) fertility\ or processes of a company to a foreign country. The answer is (B).
\ rate from the crude death rate.\nA: Let's think step by step. We refer to Wikipedia\ - question: 'Which of the following statements is NOT accurate regarding the services
\ articles on geography for help. The difference between number of births and deaths\ provided by local governments in the United States?
\ gives the population increase at any given time. The answer is (A).\n\n"
"group": "mmlu_flan_cot_fewshot_social_sciences" (A) Duplication of efforts occurs often. (B) Social problems of the central
"include": "_mmlu_flan_cot_fewshot_template_yaml" city spill over into the surrounding residential suburbs. (C) Inefficiency in
"task": "mmlu_flan_cot_fewshot_high_school_geography" providing services occurs often. (D) One neighborhood''s efforts to reduce pollution
are always supported by neighboring communities.'
target: Let's think step by step. We refer to Wikipedia articles on geography
for help. There may be economic, social or political reasons for two neighboring
communities and their local governments not agreeing to pollution reduction
efforts initiated by one of them. The answer is (D).
- question: 'The rate of natural increase of a population is found by subtracting the
(A) crude death rate from the crude birth date. (B) crude birth rate from the
crude death rate. (C) doubling time from the crude birth rate. (D) fertility
rate from the crude death rate.'
target: 'Let''s think step by step. We refer to Wikipedia articles on geography
for help. The difference between number of births and deaths gives the population
increase at any given time. The answer is (A).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_geography
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml
View file @ a2af2101
"dataset_name": "high_school_government_and_politics" dataset_name: high_school_government_and_politics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school government and politics.\n\nQ: Which of the following best states an argument\ school government and politics.
\ made by James Madison in The Federalist number 10?\n(A) Honest politicians can\ fewshot_config:
\ prevent factions from developing. (B) Factions are more likely to occur in large\ sampler: first_n
\ republics than in small ones. (C) The negative effects of factionalism can be\ samples:
\ reduced by a republican government. (D) Free elections are the people's best defense\ - question: 'Which of the following best states an argument made by James Madison in
\ against factionalism.\nA: Let's think step by step. We refer to Wikipedia articles\ The Federalist number 10?
\ on government and politics for help. In the Federalist number 10, James Madison\
\ advocated for a representative republican form of government to guard against\ (A) Honest politicians can prevent factions from developing. (B) Factions are
\ factionalism. The answer is (C).\n\nQ: The term \"budget deficit\" refers to the\n\ more likely to occur in large republics than in small ones. (C) The negative
(A) annual increase in federal spending on the military (B) amount of interest on\ effects of factionalism can be reduced by a republican government. (D) Free
\ the national debt (C) difference between the initial budget proposals made by\ elections are the people''s best defense against factionalism.'
\ the president and Congress (D) amount the government spends in excess of its revenues\n\ target: Let's think step by step. We refer to Wikipedia articles on government
A: Let's think step by step. We refer to Wikipedia articles on government and politics\ and politics for help. In the Federalist number 10, James Madison advocated
\ for help. When the goverment spends more than it earns, their difference is the\ for a representative republican form of government to guard against factionalism.
\ budget deficit. The answer is (D).\n\nQ: Which of the following statements about\ The answer is (C).
\ cabinet departments is FALSE?\n(A) They are established by the legislative branch.\ - question: 'The term "budget deficit" refers to the
\ (B) Their members often don't have much influence over presidential decisions.\
\ (C) They cannot all be run by leaders who belong to the same political party the\ (A) annual increase in federal spending on the military (B) amount of interest
\ president does. (D) Not every federal agency is a cabinet department.\nA: Let's\ on the national debt (C) difference between the initial budget proposals made
\ think step by step. We refer to Wikipedia articles on government and politics\ by the president and Congress (D) amount the government spends in excess of
\ for help. There is no law stipulating that some cabinet department leaders have\ its revenues'
\ to belong to a political party different from that of the president. The answer\ target: Let's think step by step. We refer to Wikipedia articles on government
\ is (C).\n\nQ: Which of the following cases established the precedent that a defendant\ and politics for help. When the goverment spends more than it earns, their difference
\ must be informed of the right to remain silent, the right to a lawyer, and protection\ is the budget deficit. The answer is (D).
\ from self-incrimination?\n(A) Weeks v. United States (B) Betts v. Brady (C) Mapp\ - question: 'Which of the following statements about cabinet departments is FALSE?
\ v. Ohio (D) Miranda v. Arizona\nA: Let's think step by step. We refer to Wikipedia\
\ articles on government and politics for help. In the landmark Miranda v. Arizona\ (A) They are established by the legislative branch. (B) Their members often
\ in 1966, the US Supreme Court, based on the Fifth and Sixth Amendment of the US\ don''t have much influence over presidential decisions. (C) They cannot all
\ Constitution, guaranteed a defendant's right to an attorney and protection from\ be run by leaders who belong to the same political party the president does.
\ self-incrimination. The answer is (D).\n\nQ: Uncertainty over the limits to presidential\ (D) Not every federal agency is a cabinet department.'
\ power is caused primarily by the fact that\n(A) the constitutional definition\ target: Let's think step by step. We refer to Wikipedia articles on government
\ of those powers is broad and unspecific (B) most people agree that the Constitution\ and politics for help. There is no law stipulating that some cabinet department
\ places too many limits on presidential power (C) the Supreme Court consistently\ leaders have to belong to a political party different from that of the president.
\ refuses to rule on cases concerning presidential powers (D) constitutional amendments\ The answer is (C).
\ have greatly increased presidential powers\nA: Let's think step by step. We refer\ - question: 'Which of the following cases established the precedent that a defendant
\ to Wikipedia articles on government and politics for help. The US Constitution\ must be informed of the right to remain silent, the right to a lawyer, and protection
\ is not very specific about the powers of the president, leading to uncertainty\ from self-incrimination?
\ over its limits. The answer is (A).\n\n"
"group": "mmlu_flan_cot_fewshot_social_sciences" (A) Weeks v. United States (B) Betts v. Brady (C) Mapp v. Ohio (D) Miranda v.
"include": "_mmlu_flan_cot_fewshot_template_yaml" Arizona'
"task": "mmlu_flan_cot_fewshot_high_school_government_and_politics" target: Let's think step by step. We refer to Wikipedia articles on government
and politics for help. In the landmark Miranda v. Arizona in 1966, the US Supreme
Court, based on the Fifth and Sixth Amendment of the US Constitution, guaranteed
a defendant's right to an attorney and protection from self-incrimination. The
answer is (D).
- question: 'Uncertainty over the limits to presidential power is caused primarily
by the fact that
(A) the constitutional definition of those powers is broad and unspecific (B)
most people agree that the Constitution places too many limits on presidential
power (C) the Supreme Court consistently refuses to rule on cases concerning
presidential powers (D) constitutional amendments have greatly increased presidential
powers'
target: 'Let''s think step by step. We refer to Wikipedia articles on government
and politics for help. The US Constitution is not very specific about the powers
of the president, leading to uncertainty over its limits. The answer is (A).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_government_and_politics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml
View file @ a2af2101
"dataset_name": "high_school_macroeconomics" dataset_name: high_school_macroeconomics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school macroeconomics.\n\nQ: Which of the following policies best describes supply-side\ school macroeconomics.
\ fiscal policy?\n(A) An increase in the money supply (B) Increased government spending\ fewshot_config:
\ (C) Lower taxes on research and development of new technology (D) Higher taxes\ sampler: first_n
\ on household income\nA: Let's think step by step. We refer to Wikipedia articles\ samples:
\ on macroeconomics for help. Supply-side fiscal policy stimulates the economy by\ - question: 'Which of the following policies best describes supply-side fiscal policy?
\ encouraging more production of goods and services through reduction in taxes and\
\ deregulation. The answer is (C).\n\nQ: The short-run Phillips curve indicates\ (A) An increase in the money supply (B) Increased government spending (C) Lower
\ a\n(A) direct relation between unemployment and inflation (B) direct relation\ taxes on research and development of new technology (D) Higher taxes on household
\ between price and quantity demanded (C) inverse relation between price and quantity\ income'
\ demanded (D) inverse relation between unemployment and inflation\nA: Let's think\ target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
\ step by step. We refer to Wikipedia articles on macroeconomics for help. The short-run\ for help. Supply-side fiscal policy stimulates the economy by encouraging more
\ Phillips curve shows that whenever unemployment decreases below a natural level,\ production of goods and services through reduction in taxes and deregulation.
\ the inflation starts increasing, and vice-versa. The answer is (D).\n\nQ: Holding\ The answer is (C).
\ all else equal which of the following monetary policies would be used to boost\ - question: 'The short-run Phillips curve indicates a
\ U.S. exports?\n(A) Increasing the discount rate (B) Increasing the reserve ratio\
\ (C) Buying government securities (D) Lowering tariffs\nA: Let's think step by\ (A) direct relation between unemployment and inflation (B) direct relation between
\ step. We refer to Wikipedia articles on macroeconomics for help. Buying government\ price and quantity demanded (C) inverse relation between price and quantity
\ securities leads to reduction in demand for US dollars from foreign buyers, thereby\ demanded (D) inverse relation between unemployment and inflation'
\ making it cheaper and hence making US exports more attractive. The answer is (C).\n\ target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
\nQ: A federal deficit occurs when\n(A) exports exceed imports. (B) imports exceed\ for help. The short-run Phillips curve shows that whenever unemployment decreases
\ exports. (C) federal tax collections exceed spending. (D) federal spending exceeds\ below a natural level, the inflation starts increasing, and vice-versa. The
\ federal tax revenues.\nA: Let's think step by step. We refer to Wikipedia articles\ answer is (D).
\ on macroeconomics for help. A federal deficit occurs when federal spending exceeds\ - question: 'Holding all else equal which of the following monetary policies would
\ federal income which is primarily from tax revenues. The answer is (D).\n\nQ:\ be used to boost U.S. exports?
\ Which of the following is not included in the U.S. GDP?\n(A) The U.S. military\
\ opens a new base in a foreign country with 1000 U.S. personnel. (B) Japanese consumers\ (A) Increasing the discount rate (B) Increasing the reserve ratio (C) Buying
\ buy thousands of CDs produced in the United States. (C) An American pop singer\ government securities (D) Lowering tariffs'
\ performs a sold-out concert in Paris. (D) A French theatrical production tours\ target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
\ dozens of American cities.\nA: Let's think step by step. We refer to Wikipedia\ for help. Buying government securities leads to reduction in demand for US dollars
\ articles on macroeconomics for help. The economic transactions related to the\ from foreign buyers, thereby making it cheaper and hence making US exports more
\ performance of the American pop-singer in Paris happens entirely outside the U.S.\ attractive. The answer is (C).
\ and hence is not included in the GDP numbers. The answer is (C).\n\n" - question: 'A federal deficit occurs when
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml" (A) exports exceed imports. (B) imports exceed exports. (C) federal tax collections
"task": "mmlu_flan_cot_fewshot_high_school_macroeconomics" exceed spending. (D) federal spending exceeds federal tax revenues.'
target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
for help. A federal deficit occurs when federal spending exceeds federal income
which is primarily from tax revenues. The answer is (D).
- question: 'Which of the following is not included in the U.S. GDP?
(A) The U.S. military opens a new base in a foreign country with 1000 U.S. personnel.
(B) Japanese consumers buy thousands of CDs produced in the United States. (C)
An American pop singer performs a sold-out concert in Paris. (D) A French theatrical
production tours dozens of American cities.'
target: 'Let''s think step by step. We refer to Wikipedia articles on macroeconomics
for help. The economic transactions related to the performance of the American
pop-singer in Paris happens entirely outside the U.S. and hence is not included
in the GDP numbers. The answer is (C).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_macroeconomics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml
View file @ a2af2101
"dataset_name": "high_school_mathematics" dataset_name: high_school_mathematics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school mathematics.\n\nQ: Simplify and write the result with a rational denominator:\ school mathematics.
\ $$\\sqrt{\\sqrt[3]{\\sqrt{\\frac{1}{729}}}}$$\n(A) \\frac{3\\sqrt{3}}{3} (B) \\\ fewshot_config:
frac{1}{3} (C) \\sqrt{3} (D) \\frac{\\sqrt{3}}{3}\nA: Let's think step by step.\ sampler: first_n
\ Factoring $729=3^6$ and combining the roots $\\frac{1}{2}\\frac{1}{3}\\frac{1}{2}=\\\ samples:
frac{1}{12}$, we get that $\\sqrt{\\sqrt[3]{\\sqrt{\\frac{1}{729}}}}=\\left(\\frac{1}{3^6}\\\ - question: 'Simplify and write the result with a rational denominator: $$\sqrt{\sqrt[3]{\sqrt{\frac{1}{729}}}}$$
right)^{\\frac{1}{12}}=\\frac{1}{3^{\\frac{1}{2}}}=\\frac{3}{\\sqrt{3}}$ The answer\
\ is (D).\n\nQ: Five thousand dollars compounded annually at an $x\\%$ interest\ (A) \frac{3\sqrt{3}}{3} (B) \frac{1}{3} (C) \sqrt{3} (D) \frac{\sqrt{3}}{3}'
\ rate takes six years to double. At the same interest rate, how many years will\ target: Let's think step by step. Factoring $729=3^6$ and combining the roots
\ it take $\\$300$ to grow to $\\$9600$?\n(A) 12 (B) 1 (C) 30 (D) 5\nA: Let's think\ $\frac{1}{2}\frac{1}{3}\frac{1}{2}=\frac{1}{12}$, we get that $\sqrt{\sqrt[3]{\sqrt{\frac{1}{729}}}}=\left(\frac{1}{3^6}\right)^{\frac{1}{12}}=\frac{1}{3^{\frac{1}{2}}}=\frac{3}{\sqrt{3}}$
\ step by step. To go from $\\$300$ to $\\$9600$, the value must go up by a factor\ The answer is (D).
\ of $9600/300=32=2^5$. Since at this interest rate it takes six years for it to\ - question: 'Five thousand dollars compounded annually at an $x\%$ interest rate takes
\ double, it will take $5*6=30$ years to grow to $\\$9600$. The answer is (C).\n\ six years to double. At the same interest rate, how many years will it take
\nQ: Ten students take a biology test and receive the following scores: 45, 55,\ $\$300$ to grow to $\$9600$?
\ 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students’ test scores?\n\
(A) 55 (B) 60 (C) 62 (D) 65\nA: Let's think step by step. There are 10 students\ (A) 12 (B) 1 (C) 30 (D) 5'
\ and the sum of their scores is $45 + 55 + 50 + 70 + 65 + 80 + 40 + 90 + 70 + 85\ target: Let's think step by step. To go from $\$300$ to $\$9600$, the value must
\ = 650$, the mean is $650/10=65$. The answer is (D).\n\nQ: The variable $x$ varies\ go up by a factor of $9600/300=32=2^5$. Since at this interest rate it takes
\ directly as the square of $y$, and $y$ varies directly as the cube of $z$. If\ six years for it to double, it will take $5*6=30$ years to grow to $\$9600$.
\ $x$ equals $-16$ when $z$ equals 2, what is the value of $x$ when $z$ equals $\\\ The answer is (C).
frac{1}{2}$?\n(A) -1 (B) 16 (C) -\\frac{1}{256} (D) \\frac{1}{16}\nA: Let's think\ - question: "Ten students take a biology test and receive the following scores: 45,\
\ step by step. We know that $x \\propto y^2$ and $y \\propto z^3$, so $x = k z^6$\ \ 55, 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students\u2019\
\ for some constant $k$. Plugging in for $x=-16$ and $z=2$, the constant value is\ \ test scores?\n(A) 55 (B) 60 (C) 62 (D) 65"
\ $k=\\frac{x}{z^6}=\\frac{-16}{64}=-\\frac{1}{4}$. So, when $z=\\frac{1}{2}$, the\ target: Let's think step by step. There are 10 students and the sum of their scores
\ value of $x$ is $x=kz^6=-\\frac{1}{4}\\frac{1}{2^6}=-\\frac{1}{256}$. The answer\ is $45 + 55 + 50 + 70 + 65 + 80 + 40 + 90 + 70 + 85 = 650$, the mean is $650/10=65$.
\ is (C).\n\nQ: Joe was in charge of lights for a dance. The red light blinks every\ The answer is (D).
\ two seconds, the yellow light every three seconds, and the blue light every five\ - question: 'The variable $x$ varies directly as the square of $y$, and $y$ varies
\ seconds. If we include the very beginning and very end of the dance, how many\ directly as the cube of $z$. If $x$ equals $-16$ when $z$ equals 2, what is
\ times during a seven minute dance will all the lights come on at the same time?\ the value of $x$ when $z$ equals $\frac{1}{2}$?
\ (Assume that all three lights blink simultaneously at the very beginning of the\
\ dance.)\n(A) 3 (B) 15 (C) 6 (D) 5\nA: Let's think step by step. The least common\ (A) -1 (B) 16 (C) -\frac{1}{256} (D) \frac{1}{16}'
\ multiple of 2, 3 and 5 is 30, so during a 7 minute dance, all the three lights\ target: Let's think step by step. We know that $x \propto y^2$ and $y \propto
\ will come on at the same time $2*7+1=15$ times. The answer is (B).\n\n" z^3$, so $x = k z^6$ for some constant $k$. Plugging in for $x=-16$ and $z=2$,
"group": "mmlu_flan_cot_fewshot_stem" the constant value is $k=\frac{x}{z^6}=\frac{-16}{64}=-\frac{1}{4}$. So, when
"include": "_mmlu_flan_cot_fewshot_template_yaml" $z=\frac{1}{2}$, the value of $x$ is $x=kz^6=-\frac{1}{4}\frac{1}{2^6}=-\frac{1}{256}$.
"task": "mmlu_flan_cot_fewshot_high_school_mathematics" The answer is (C).
- question: 'Joe was in charge of lights for a dance. The red light blinks every two
seconds, the yellow light every three seconds, and the blue light every five
seconds. If we include the very beginning and very end of the dance, how many
times during a seven minute dance will all the lights come on at the same time?
(Assume that all three lights blink simultaneously at the very beginning of
the dance.)
(A) 3 (B) 15 (C) 6 (D) 5'
target: 'Let''s think step by step. The least common multiple of 2, 3 and 5 is
30, so during a 7 minute dance, all the three lights will come on at the same
time $2*7+1=15$ times. The answer is (B).'
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_mathematics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml
View file @ a2af2101
"dataset_name": "high_school_microeconomics" dataset_name: high_school_microeconomics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school microeconomics.\n\nQ: Which of the following is necessarily a characteristic\ school microeconomics.
\ of oligopoly?\n(A) Free entry into and exit from the market (B) A few large producers\ fewshot_config:
\ (C) One producer of a good with no close substitutes (D) A homogenous product\n\ sampler: first_n
A: Let's think step by step. We refer to Wikipedia articles on microeconomics for\ samples:
\ help. An oligopoly is when a market is dominated by just one or a few number of\ - question: 'Which of the following is necessarily a characteristic of oligopoly?
\ sellers or producers. To get oligopoly, the market should have high barriers to\
\ new entry, and the product has differentiation. The answer is (B).\n\nQ: If the\ (A) Free entry into and exit from the market (B) A few large producers (C) One
\ government subsidizes producers in a perfectly competitive market, then\n(A) the\ producer of a good with no close substitutes (D) A homogenous product'
\ demand for the product will increase (B) the demand for the product will decrease\ target: Let's think step by step. We refer to Wikipedia articles on microeconomics
\ (C) the consumer surplus will increase (D) the consumer surplus will decrease\n\ for help. An oligopoly is when a market is dominated by just one or a few number
A: Let's think step by step. We refer to Wikipedia articles on microeconomics for\ of sellers or producers. To get oligopoly, the market should have high barriers
\ help. (A) and (B) are wrong because the demand curve does not change at all. If\ to new entry, and the product has differentiation. The answer is (B).
\ the government subsidizes producers, the supply will increase, and thus the consumer\ - question: 'If the government subsidizes producers in a perfectly competitive market,
\ surplus also increases. The answer is (C).\n\nQ: Which of the following is true\ then
\ of a price floor?\n(A) The price floor shifts the demand curve to the left. (B)\
\ An effective floor creates a shortage of the good. (C) The price floor shifts\ (A) the demand for the product will increase (B) the demand for the product
\ the supply curve of the good to the right. (D) To be an effective floor, it must\ will decrease (C) the consumer surplus will increase (D) the consumer surplus
\ be set above the equilibrium price.\nA: Let's think step by step. We refer to\ will decrease'
\ Wikipedia articles on microeconomics for help. Price floor does not shift the\ target: Let's think step by step. We refer to Wikipedia articles on microeconomics
\ demand or shift curve. An effective price floor should be set above the equilibrium\ for help. (A) and (B) are wrong because the demand curve does not change at
\ price, otherwise the market bears and the floor does not have effective effect.\ all. If the government subsidizes producers, the supply will increase, and thus
\ The answer is (D).\n\nQ: The concentration ratio for a monopoly is\n(A) 0 (B)\ the consumer surplus also increases. The answer is (C).
\ 5 (C) 10 (D) 100\nA: Let's think step by step. We refer to Wikipedia articles\ - question: 'Which of the following is true of a price floor?
\ on microeconomics for help. The concentration ratio is calculated as the sum of\
\ market share of a specific number of largest companies. Monopoly means one company\ (A) The price floor shifts the demand curve to the left. (B) An effective floor
\ or entity controls the entire market, therefore, the concentration ratio is 100\ creates a shortage of the good. (C) The price floor shifts the supply curve
\ percent. The answer is (D).\n\nQ: In a competitive labor market for housepainters,\ of the good to the right. (D) To be an effective floor, it must be set above
\ which of the following would increase the demand for housepainters?\n(A) An effective\ the equilibrium price.'
\ minimum wage imposed on this labor market. (B) An increase in the price of gallons\ target: Let's think step by step. We refer to Wikipedia articles on microeconomics
\ of paint. (C) An increase in the construction of new houses. (D) An increase in\ for help. Price floor does not shift the demand or shift curve. An effective
\ the price of mechanical painters so long as the output effect exceeds the substitution\ price floor should be set above the equilibrium price, otherwise the market
\ effect.\nA: Let's think step by step. We refer to Wikipedia articles on microeconomics\ bears and the floor does not have effective effect. The answer is (D).
\ for help. An increase in the construction of new houses means an increase demand\ - question: 'The concentration ratio for a monopoly is
\ of in-house painting, thus increases the demand for housepainters. The answer\
\ is (C).\n\n" (A) 0 (B) 5 (C) 10 (D) 100'
"group": "mmlu_flan_cot_fewshot_social_sciences" target: Let's think step by step. We refer to Wikipedia articles on microeconomics
"include": "_mmlu_flan_cot_fewshot_template_yaml" for help. The concentration ratio is calculated as the sum of market share of
"task": "mmlu_flan_cot_fewshot_high_school_microeconomics" a specific number of largest companies. Monopoly means one company or entity
controls the entire market, therefore, the concentration ratio is 100 percent.
The answer is (D).
- question: 'In a competitive labor market for housepainters, which of the following
would increase the demand for housepainters?
(A) An effective minimum wage imposed on this labor market. (B) An increase
in the price of gallons of paint. (C) An increase in the construction of new
houses. (D) An increase in the price of mechanical painters so long as the output
effect exceeds the substitution effect.'
target: 'Let''s think step by step. We refer to Wikipedia articles on microeconomics
for help. An increase in the construction of new houses means an increase demand
of in-house painting, thus increases the demand for housepainters. The answer
is (C).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_microeconomics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml
View file @ a2af2101
"dataset_name": "high_school_physics" dataset_name: high_school_physics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school physics.\n\nQ: A microwave oven is connected to an outlet, 120 V, and draws\ school physics.
\ a current of 2 amps. At what rate is energy being used by the microwave oven?\n\ fewshot_config:
(A) 10 W (B) 30 W (C) 60 W (D) 240 W\nA: Let's think step by step. Rate of energy\ sampler: first_n
\ usage is known as power; in an dissipative electrical circuit, power is given\ samples:
\ by voltage times current. So in our case, the power is 120 V times 2 amps, or\ - question: 'A microwave oven is connected to an outlet, 120 V, and draws a current
\ 240 W. The answer is (D).\n\nQ: A point charge, Q = +1 mC, is fixed at the origin.\ of 2 amps. At what rate is energy being used by the microwave oven?
\ How much work is required to move a charge, Q = +8 µC, from the point (0, 4 meters)\
\ to the point (3 meters, 0)?\n(A) 3.5 J (B) 6.0 J (C) 22.5 J (D) 40 J\nA: Let's\ (A) 10 W (B) 30 W (C) 60 W (D) 240 W'
\ think step by step. To calculate the work required to move a charge from one location\ target: Let's think step by step. Rate of energy usage is known as power; in an
\ to another in a fixed electric field, it is enough to calculate the potential\ dissipative electrical circuit, power is given by voltage times current. So
\ difference between the two locations. Here, the potential only depends on the\ in our case, the power is 120 V times 2 amps, or 240 W. The answer is (D).
\ distance between the charges; it’s $k q_1 q_2 / r$, where $k$ is Coulomb’s constant.\ - question: "A point charge, Q = +1 mC, is fixed at the origin. How much work is required\
\ Plugging in values $q_1 = $ 1 mC, $q_2 = 8 \\mu$ C, gives the answer as 5.992\ \ to move a charge, Q = +8 \xB5C, from the point (0, 4 meters) to the point\
\ J, which rounds to 6 J. The answer is (B).\n\nQ: Which of the following conditions\ \ (3 meters, 0)?\n(A) 3.5 J (B) 6.0 J (C) 22.5 J (D) 40 J"
\ will ensure that angular momentum is conserved? I. Conservation of linear momentum\ target: "Let's think step by step. To calculate the work required to move a charge\
\ II. Zero net external force III. Zero net external torque\n(A) I and II only (B)\ \ from one location to another in a fixed electric field, it is enough to calculate\
\ I and III only (C) II and III only (D) III only\nA: Let's think step by step.\ \ the potential difference between the two locations. Here, the potential only\
\ Torque is defined as the change in angular momentum; if there is zero external\ \ depends on the distance between the charges; it\u2019s $k q_1 q_2 / r$, where\
\ torque, angular momentum is conserved. The answer is (D).\n\nQ: A photocell of\ \ $k$ is Coulomb\u2019s constant. Plugging in values $q_1 = $ 1 mC, $q_2 = 8\
\ work function ϕ = 2eV is connected to a resistor in series. Light of frequency\ \ \\mu$ C, gives the answer as 5.992 J, which rounds to 6 J. The answer is (B)."
\ f = 1 × 10^15 Hz hits a metal plate of the photocell. If the power of the light\ - question: 'Which of the following conditions will ensure that angular momentum is
\ is P = 100 W, what is the current through the resistor?\n(A) 2:00 AM (B) 6:00\ conserved? I. Conservation of linear momentum II. Zero net external force III.
\ AM (C) 12:00 AM (D) 24 A\nA: Let's think step by step. The only answer above which\ Zero net external torque
\ has units of current is D, 24 A. The answer is (D).\n\nQ: A pipe full of air is\
\ closed at one end. A standing wave is produced in the pipe, causing the pipe to\ (A) I and II only (B) I and III only (C) II and III only (D) III only'
\ sound a note. Which of the following is a correct statement about the wave’s properties\ target: Let's think step by step. Torque is defined as the change in angular momentum;
\ at the closed end of the pipe?\n(A) The pressure is at a node, but the particle\ if there is zero external torque, angular momentum is conserved. The answer
\ displacement is at an antinode. (B) The pressure is at an antinode, but the particle\ is (D).
\ displacement is at a node. (C) The pressure and the particle displacement are\ - question: "A photocell of work function \u03D5 = 2eV is connected to a resistor in\
\ both at nodes. (D) The pressure and the particle displacement are both at antinodes.\n\ \ series. Light of frequency f = 1 \xD7 10^15 Hz hits a metal plate of the photocell.\
A: Let's think step by step. At the closed end of the pipe, the particles cannot\ \ If the power of the light is P = 100 W, what is the current through the resistor?\n\
\ have any net displacement because the pipe closure stops them. So the particle\ (A) 2:00 AM (B) 6:00 AM (C) 12:00 AM (D) 24 A"
\ displacement is at a node. This closure also causes the pressure to be maximal,\ target: Let's think step by step. The only answer above which has units of current
\ i.e. an antinode. The answer is (B).\n\n" is D, 24 A. The answer is (D).
"group": "mmlu_flan_cot_fewshot_stem" - question: "A pipe full of air is closed at one end. A standing wave is produced in\
"include": "_mmlu_flan_cot_fewshot_template_yaml" \ the pipe, causing the pipe to sound a note. Which of the following is a correct\
"task": "mmlu_flan_cot_fewshot_high_school_physics" \ statement about the wave\u2019s properties at the closed end of the pipe?\n\
(A) The pressure is at a node, but the particle displacement is at an antinode.\
\ (B) The pressure is at an antinode, but the particle displacement is at a\
\ node. (C) The pressure and the particle displacement are both at nodes. (D)\
\ The pressure and the particle displacement are both at antinodes."
target: 'Let''s think step by step. At the closed end of the pipe, the particles
cannot have any net displacement because the pipe closure stops them. So the
particle displacement is at a node. This closure also causes the pressure to
be maximal, i.e. an antinode. The answer is (B).'
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_physics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml
View file @ a2af2101
"dataset_name": "high_school_psychology" dataset_name: high_school_psychology
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school psychology.\n\nQ: Pascale is interested in the processing strategies children\ school psychology.
\ use to learn new information. Pascale would best be classified as what type of\ fewshot_config:
\ psychologist?\n(A) sociocultural (B) clinical (C) cognitive (D) behaviorist\n\ sampler: first_n
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\ samples:
\ Sociocultural psychologist focuses on the effect of societal factors on people.\ - question: 'Pascale is interested in the processing strategies children use to learn
\ Clinical psychologist focuses on people with mental issues. Cognitive psychologist\ new information. Pascale would best be classified as what type of psychologist?
\ focuses on how people think and learn, including the processing strategies. Behaviorist\
\ focuses more on the environment and experience effect on people. The answer is\ (A) sociocultural (B) clinical (C) cognitive (D) behaviorist'
\ (C).\n\nQ: According to Caplan's model of consultee-centered case consultation,\ target: Let's think step by step. We refer to Wikipedia articles on psychology
\ the consultant is primarily interested in\n(A) identifying the causes and solutions\ for help. Sociocultural psychologist focuses on the effect of societal factors
\ of the client's presenting problems (B) identifying and eliminating the causes\ on people. Clinical psychologist focuses on people with mental issues. Cognitive
\ of the consultee's difficulties in handling a problem (C) establishing a hierarchy\ psychologist focuses on how people think and learn, including the processing
\ of authority to enable effective decision making (D) presenting a single, well-defined\ strategies. Behaviorist focuses more on the environment and experience effect
\ and unambiguous course of action for the consultant to overcome skills deficits\n\ on people. The answer is (C).
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\ - question: 'According to Caplan''s model of consultee-centered case consultation,
\ Caplan defines two type of consultation. Client-centered case consultation aims\ the consultant is primarily interested in
\ to handle client's problems, while consultee-centered case consultation aims to\
\ identify the reason of client's difficulty to solve problems. The answer is (B).\n\ (A) identifying the causes and solutions of the client''s presenting problems
\nQ: According to the Individuals with Disabilities Education Improvement Act, which\ (B) identifying and eliminating the causes of the consultee''s difficulties
\ of the following must an educational agency do before it changes the educational\ in handling a problem (C) establishing a hierarchy of authority to enable effective
\ placement of a student with a disability?\n(A) Give the child a trial period in\ decision making (D) presenting a single, well-defined and unambiguous course
\ the new environment (B) Notify the parents in writing (C) Obtain school board\ of action for the consultant to overcome skills deficits'
\ approval (D) Obtain parental consent\nA: Let's think step by step. We refer to\ target: Let's think step by step. We refer to Wikipedia articles on psychology
\ Wikipedia articles on psychology for help. When the decision to change the educational\ for help. Caplan defines two type of consultation. Client-centered case consultation
\ placement of a student with a disability is made, the educational agency must\ aims to handle client's problems, while consultee-centered case consultation
\ notify the parents in writing on that date. The answer is (B).\n\nQ: While swimming\ aims to identify the reason of client's difficulty to solve problems. The answer
\ in the ocean, Ivan is frightened by a dark shadow in the water even before he\ is (B).
\ has the chance to identify what the shadow is. The synaptic connections taking\ - question: 'According to the Individuals with Disabilities Education Improvement Act,
\ place during this incident of fright are best described by which of the following?\n\ which of the following must an educational agency do before it changes the educational
(A) Messages are sent from the thalamus directly to the amygdala. (B) Messages are\ placement of a student with a disability?
\ sent from the thalamus to the \"what\" and \"where\" pathways. (C) Messages are\
\ sent from the parasympathetic nervous system to the cerebral cortex. (D) Messages\ (A) Give the child a trial period in the new environment (B) Notify the parents
\ are sent from the frontal lobes to the pituitary gland.\nA: Let's think step by\ in writing (C) Obtain school board approval (D) Obtain parental consent'
\ step. We refer to Wikipedia articles on psychology for help. Our neural system\ target: Let's think step by step. We refer to Wikipedia articles on psychology
\ has a mechanism that can respond immediate emotional signal before going to the\ for help. When the decision to change the educational placement of a student
\ thought center. In the Ivan's case, messages travel directly from thalamus to\ with a disability is made, the educational agency must notify the parents in
\ amygdala. The answer is (A).\n\nQ: Ani believes that her attitudes and behavior\ writing on that date. The answer is (B).
\ play a central role in what happens to her. Such a belief is likely to be associated\ - question: 'While swimming in the ocean, Ivan is frightened by a dark shadow in the
\ with\n(A) a strong superego. (B) low self-esteem. (C) low self-efficacy. (D) an\ water even before he has the chance to identify what the shadow is. The synaptic
\ internal locus of control.\nA: Let's think step by step. We refer to Wikipedia\ connections taking place during this incident of fright are best described by
\ articles on psychology for help. People with an external locus of control believes\ which of the following?
\ fate and luck play an important role in their lives, while people with an internal\
\ locus of control believes they control their lives. The answer is (D).\n\n" (A) Messages are sent from the thalamus directly to the amygdala. (B) Messages
"group": "mmlu_flan_cot_fewshot_social_sciences" are sent from the thalamus to the "what" and "where" pathways. (C) Messages
"include": "_mmlu_flan_cot_fewshot_template_yaml" are sent from the parasympathetic nervous system to the cerebral cortex. (D)
"task": "mmlu_flan_cot_fewshot_high_school_psychology" Messages are sent from the frontal lobes to the pituitary gland.'
target: Let's think step by step. We refer to Wikipedia articles on psychology
for help. Our neural system has a mechanism that can respond immediate emotional
signal before going to the thought center. In the Ivan's case, messages travel
directly from thalamus to amygdala. The answer is (A).
- question: 'Ani believes that her attitudes and behavior play a central role in what
happens to her. Such a belief is likely to be associated with
(A) a strong superego. (B) low self-esteem. (C) low self-efficacy. (D) an internal
locus of control.'
target: 'Let''s think step by step. We refer to Wikipedia articles on psychology
for help. People with an external locus of control believes fate and luck play
an important role in their lives, while people with an internal locus of control
believes they control their lives. The answer is (D).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_psychology
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml
View file @ a2af2101
"dataset_name": "high_school_statistics" dataset_name: high_school_statistics
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school statistics.\n\nQ: A new smartwatch is manufactured in one part of a factory,\ school statistics.
\ then secured for shipping in another, independent part of the factory. The weight\ fewshot_config:
\ of the smartwatch has a mean of 62 grams and a standard deviation of 1.0 grams.\ sampler: first_n
\ The weight of the packaging (box, user's guide, bubble wrap, etc.) has a mean\ samples:
\ of 456 grams and a standard deviation of 6 grams. Together, the distribution of\ - question: 'A new smartwatch is manufactured in one part of a factory, then secured
\ the weight of the smartwatch and its packaging would have the following mean and\ for shipping in another, independent part of the factory. The weight of the
\ standard deviation:\n(A) Mean 518 grams; standard deviation 7.0 grams (B) Mean\ smartwatch has a mean of 62 grams and a standard deviation of 1.0 grams. The
\ 518 grams; standard deviation 3.5 grams (C) Mean 518 grams; standard deviation\ weight of the packaging (box, user''s guide, bubble wrap, etc.) has a mean of
\ 6.1 grams (D) Mean 394 grams; standard deviation 6.1 grams\nA: Let's think step\ 456 grams and a standard deviation of 6 grams. Together, the distribution of
\ by step. Since the weight of the watch and the weight of the packaging are independent\ the weight of the smartwatch and its packaging would have the following mean
\ random variables, the mean and variance of their sum is equal to the sum of their\ and standard deviation:
\ individual means and variances. So the mean is 62 + 456 = 518 grams, and the variances\
\ is 1.0^2 + 6.0^2 = 37, leading to a standard deviation of 6.1 grams. The answer\ (A) Mean 518 grams; standard deviation 7.0 grams (B) Mean 518 grams; standard
\ is (C).\n\nQ: After a frost warning was issued, the owner of a large orange grove\ deviation 3.5 grams (C) Mean 518 grams; standard deviation 6.1 grams (D) Mean
\ asked his workers to spray all his trees with water. The water was supposed to\ 394 grams; standard deviation 6.1 grams'
\ freeze and form a protective covering of ice around the orange blossom. Nevertheless,\ target: Let's think step by step. Since the weight of the watch and the weight
\ the owner suspected that some trees suffered considerable damage due to the frost.\ of the packaging are independent random variables, the mean and variance of
\ To estimate the proportion of trees that suffered more than 50 percent damage\ their sum is equal to the sum of their individual means and variances. So the
\ due to the frost, he took a random sample of 100 trees from his grove. What is\ mean is 62 + 456 = 518 grams, and the variances is 1.0^2 + 6.0^2 = 37, leading
\ the response variable in this experiment?\n(A) The proportion of trees that suffered\ to a standard deviation of 6.1 grams. The answer is (C).
\ more than 50 percent damage due to frost. (B) The number of trees affected by\ - question: 'After a frost warning was issued, the owner of a large orange grove asked
\ the frost. (C) The number of trees sampled from the grove. (D) For each sampled\ his workers to spray all his trees with water. The water was supposed to freeze
\ tree, whether it suffered more than 50 percent damage or at most 50 percent damage.\n\ and form a protective covering of ice around the orange blossom. Nevertheless,
A: Let's think step by step. In this experiment, the response variable is what is\ the owner suspected that some trees suffered considerable damage due to the
\ measured. For each tree, what is measured is whether or not it suffered more than\ frost. To estimate the proportion of trees that suffered more than 50 percent
\ 50 percent damage due to the frost. The answer is (D).\n\nQ: Suppose X and Y are\ damage due to the frost, he took a random sample of 100 trees from his grove.
\ random variables with E(X) = 37, var(X) = 5, E(Y) = 62, and var(Y) = 12. What\ What is the response variable in this experiment?
\ are the expected value and variance of the random variable X + Y?\n(A) E(X + Y)\
\ = 99, var(X + Y) = 8.5 (B) E(X + Y) = 99, var(X + Y) = 13 (C) E(X + Y) = 99, var(X\ (A) The proportion of trees that suffered more than 50 percent damage due to
\ + Y) = 17 (D) There is insufficient information to answer this question.\nA: Let's\ frost. (B) The number of trees affected by the frost. (C) The number of trees
\ think step by step. While means of sums of random variables add (regardless of\ sampled from the grove. (D) For each sampled tree, whether it suffered more
\ whether the variables are independent) in order to determine the variance of a\ than 50 percent damage or at most 50 percent damage.'
\ sum of random variables, we need to know not just their individual variances but\ target: Let's think step by step. In this experiment, the response variable is
\ the covariance of the two variables, which is not given in this problem. The answer\ what is measured. For each tree, what is measured is whether or not it suffered
\ is (D).\n\nQ: Which of the following sets has the smallest standard deviation?\ more than 50 percent damage due to the frost. The answer is (D).
\ Which has the largest?\nI: {1,2,3}\nII: {-10,10}\nIII: {100}\n(A) I, II (B) II,\ - question: 'Suppose X and Y are random variables with E(X) = 37, var(X) = 5, E(Y)
\ III (C) III, I (D) III, II\nA: Let's think step by step. The variance of distribution\ = 62, and var(Y) = 12. What are the expected value and variance of the random
\ I is the expected squared deviation from its mean (which is 2), so the variance\ variable X + Y?
\ is 2/3 . The variance of distribution II is 10^2 (because both elements are 10\
\ away from the mean of zero). The variance of distribution III is 0, since it has\ (A) E(X + Y) = 99, var(X + Y) = 8.5 (B) E(X + Y) = 99, var(X + Y) = 13 (C) E(X
\ a single entry. So distribution III has the smallest standard deviation and distribution\ + Y) = 99, var(X + Y) = 17 (D) There is insufficient information to answer this
\ II has the largest. The answer is (D).\n\nQ: Which of the following is a correct\ question.'
\ statement about correlation?\n(A) If the slope of the regression line is exactly\ target: Let's think step by step. While means of sums of random variables add
\ 1, then the correlation is exactly 1. (B) If the correlation is 0, then the slope\ (regardless of whether the variables are independent) in order to determine
\ of the regression line is undefined. (C) Switching which variable is called x\ the variance of a sum of random variables, we need to know not just their individual
\ and which is called y changes the sign of the correlation. (D) The correlation\ variances but the covariance of the two variables, which is not given in this
\ r is equal to the slope of the regression line when z-scores for the y-variable\ problem. The answer is (D).
\ are plotted against z-scores for the x-variable.\nA: Let's think step by step.\ - question: 'Which of the following sets has the smallest standard deviation? Which
\ Statement A is false because the slope of the regression line being exactly 1\ has the largest?
\ can occur even when the two variables are not perfectly correlated. Statement\
\ B is false because uncorrelated variables regression lines can have slope zero.\ I: {1,2,3}
\ Statement C is false because correlation is symmetric in the two random variables.\
\ The answer is (D).\n\n" II: {-10,10}
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml" III: {100}
"task": "mmlu_flan_cot_fewshot_high_school_statistics"
(A) I, II (B) II, III (C) III, I (D) III, II'
target: Let's think step by step. The variance of distribution I is the expected
squared deviation from its mean (which is 2), so the variance is 2/3 . The variance
of distribution II is 10^2 (because both elements are 10 away from the mean
of zero). The variance of distribution III is 0, since it has a single entry.
So distribution III has the smallest standard deviation and distribution II
has the largest. The answer is (D).
- question: 'Which of the following is a correct statement about correlation?
(A) If the slope of the regression line is exactly 1, then the correlation is
exactly 1. (B) If the correlation is 0, then the slope of the regression line
is undefined. (C) Switching which variable is called x and which is called y
changes the sign of the correlation. (D) The correlation r is equal to the slope
of the regression line when z-scores for the y-variable are plotted against
z-scores for the x-variable.'
target: 'Let''s think step by step. Statement A is false because the slope of
the regression line being exactly 1 can occur even when the two variables are
not perfectly correlated. Statement B is false because uncorrelated variables
regression lines can have slope zero. Statement C is false because correlation
is symmetric in the two random variables. The answer is (D).'
tag: mmlu_flan_cot_fewshot_stem
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_statistics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml
View file @ a2af2101
"dataset_name": "high_school_us_history" dataset_name: high_school_us_history
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school us history.\n\nQ: This question refers to the following information.\n\ school us history.
I come not to urge personal claims, nor to seek individual benefits; I appear as\ fewshot_config:
\ the advocate of those who cannot plead their own cause; I come as the friend of\ sampler: first_n
\ those who are deserted, oppressed, and desolate. In the Providence of God, I am\ samples:
\ the voice of the maniac whose piercing cries from the dreary dungeons of your\ - question: "This question refers to the following information.\nI come not to urge\
\ jails penetrate not your Halls of Legislation. I am the Hope of the poor crazed\ \ personal claims, nor to seek individual benefits; I appear as the advocate\
\ beings who pine in the cells, and stalls, and cages, and waste rooms of your poor-houses.\ \ of those who cannot plead their own cause; I come as the friend of those who\
\ I am the Revelation of hundreds of wailing, suffering creatures, hidden in your\ \ are deserted, oppressed, and desolate. In the Providence of God, I am the\
\ private dwellings, and in pens and cabins—shut out, cut off from all healing influences,\ \ voice of the maniac whose piercing cries from the dreary dungeons of your\
\ from all mind-restoring cares.… Could their melancholy histories be spread before\ \ jails penetrate not your Halls of Legislation. I am the Hope of the poor crazed\
\ you as revealed to my grieved spirit during the last three months, how promptly,\ \ beings who pine in the cells, and stalls, and cages, and waste rooms of your\
\ how earnestly would you search out the most approved means of relief; how trifling,\ \ poor-houses. I am the Revelation of hundreds of wailing, suffering creatures,\
\ how insignificant, by comparison, would appear the sacrifices you are asked to\ \ hidden in your private dwellings, and in pens and cabins\u2014shut out, cut\
\ make; how would a few dimes and dollars, gathered from each citizen, diminish\ \ off from all healing influences, from all mind-restoring cares.\u2026 Could\
\ in value as a possession, compared with the certain benefits and vast good to\ \ their melancholy histories be spread before you as revealed to my grieved\
\ be secured for the suffering insane...by the consecration and application of a\ \ spirit during the last three months, how promptly, how earnestly would you\
\ sufficient fund to the construction of a suitable hospital.…\n—Dorothea Dix, Memorial\ \ search out the most approved means of relief; how trifling, how insignificant,\
\ Soliciting a State Hospital for the Protection and Cure of the Insane,\nSubmitted\ \ by comparison, would appear the sacrifices you are asked to make; how would\
\ to the General Assembly of North Carolina, November 1848\nDorothea Dix can best\ \ a few dimes and dollars, gathered from each citizen, diminish in value as\
\ be compared to whom?\n(A) Abigail Adams (B) Clara Barton (C) Shirley Temple (D)\ \ a possession, compared with the certain benefits and vast good to be secured\
\ Hillary Clinton\nA: Let's think step by step. We refer to Wikipedia articles on\ \ for the suffering insane...by the consecration and application of a sufficient\
\ us history for help. Both Dorothea Dix and Clara barton are American nurses. The\ \ fund to the construction of a suitable hospital.\u2026\n\u2014Dorothea Dix,\
\ answer is (B).\n\nQ: This question refers to the following information.\n\"As\ \ Memorial Soliciting a State Hospital for the Protection and Cure of the Insane,\n\
\ our late Conduct at the Conestoga Manor and Lancaster have occasioned much Speculation\ Submitted to the General Assembly of North Carolina, November 1848\nDorothea\
\ & a great diversity of Sentiments in this and neighboring Governments; some vindicating\ \ Dix can best be compared to whom?\n(A) Abigail Adams (B) Clara Barton (C)\
\ & others condemning it; some charitably alleviating the Crime, & others maliciously\ \ Shirley Temple (D) Hillary Clinton"
\ painting it in the most odious & detestable Colours, we think it our duty to lay\ target: Let's think step by step. We refer to Wikipedia articles on us history
\ before the Publick, the whole Matter as it appeared, & still appears, to us. .\ for help. Both Dorothea Dix and Clara barton are American nurses. The answer
\ . .\n\"If these things are not sufficient to prove an unjustifiable Attachment\ is (B).
\ in the Quakers to the Indians Savages, a fixed Resolution to befriend them & an\ - question: "This question refers to the following information.\n\"As our late Conduct\
\ utter insensibility to human Distresses, let us consider a few more recent Facts.\ \ at the Conestoga Manor and Lancaster have occasioned much Speculation & a\
\ When we found the last Summer that we were likely to get no Assistance from the\ \ great diversity of Sentiments in this and neighboring Governments; some vindicating\
\ Government, some Volunteers went out at our own Expense, determined to drive our\ \ & others condemning it; some charitably alleviating the Crime, & others maliciously\
\ Enemies from our Borders; & when we came near to the great Island, we understood\ \ painting it in the most odious & detestable Colours, we think it our duty\
\ that a Number of their Warriors had gone out against our Frontiers. Upon this\ \ to lay before the Publick, the whole Matter as it appeared, & still appears,\
\ we returned and came up with them and fought with them at the Munfey Hill where\ \ to us. . . .\n\"If these things are not sufficient to prove an unjustifiable\
\ we lost some of our Men & killed some of their Warriors & thereby saved our Frontiers\ \ Attachment in the Quakers to the Indians Savages, a fixed Resolution to befriend\
\ from this Story in another Expedition. But no sooner had we destroyed their Provisions\ \ them & an utter insensibility to human Distresses, let us consider a few more\
\ on the great Island, & ruined their trade with the good People at Bethlehem, but\ \ recent Facts. When we found the last Summer that we were likely to get no\
\ these very Indians, who were justly suspected of having murdered our Friends in\ \ Assistance from the Government, some Volunteers went out at our own Expense,\
\ Northampton County, were by the Influence of some Quakers taken under the Protection\ \ determined to drive our Enemies from our Borders; & when we came near to the\
\ of the Government to screen them from the Resentments of the Friends and Relations\ \ great Island, we understood that a Number of their Warriors had gone out against\
\ of the Murdered, & to support them thro the Winter.\"\n—\"Apology of the Paxton\ \ our Frontiers. Upon this we returned and came up with them and fought with\
\ Boys\" (pamphlet), 1764 (Note: \"apology\" in this context should be read as an\ \ them at the Munfey Hill where we lost some of our Men & killed some of their\
\ explanation, not an admission of guilt or regret.\nThe sentiments expressed in\ \ Warriors & thereby saved our Frontiers from this Story in another Expedition.\
\ the explanation above reflect which of the ongoing tensions during the colonial\ \ But no sooner had we destroyed their Provisions on the great Island, & ruined\
\ period of American history?\n(A) Tensions between British policies and the aspirations\ \ their trade with the good People at Bethlehem, but these very Indians, who\
\ of North American colonists. (B) Tensions between American Indians allied with\ \ were justly suspected of having murdered our Friends in Northampton County,\
\ the French and those allied with the British. (C) Tensions between freed African\ \ were by the Influence of some Quakers taken under the Protection of the Government\
\ Americans and white planters. (D) Tensions between backcountry settlers and elites\ \ to screen them from the Resentments of the Friends and Relations of the Murdered,\
\ within colonial America.\nA: Let's think step by step. We refer to Wikipedia articles\ \ & to support them thro the Winter.\"\n\u2014\"Apology of the Paxton Boys\"\
\ on us history for help. After the French and Indian War, the Scotch-Irish settlers\ \ (pamphlet), 1764 (Note: \"apology\" in this context should be read as an explanation,\
\ attacked American Indians. After the attacks on the Conestoga, about 250 Paxton\ \ not an admission of guilt or regret.\nThe sentiments expressed in the explanation\
\ Boys present their grievances to the Pennsylvania legislature. As mentioned in\ \ above reflect which of the ongoing tensions during the colonial period of\
\ the information, the Paxton Boys cited resentiment at local elites. The answer\ \ American history?\n(A) Tensions between British policies and the aspirations\
\ is (D).\n\nQ: This question refers to the following information.\nOur leaders\ \ of North American colonists. (B) Tensions between American Indians allied\
\ talk about stopping aggression from the north, but this was a struggle among groups\ \ with the French and those allied with the British. (C) Tensions between freed\
\ of Vietnamese until we intervened. We seem bent upon saving the Vietnamese from\ \ African Americans and white planters. (D) Tensions between backcountry settlers\
\ Ho Chi Minh even if we have to kill them and demolish their country to do it.\ \ and elites within colonial America."
\ As the native people survey bombed-out villages, women and children burned by\ target: Let's think step by step. We refer to Wikipedia articles on us history
\ napalm, rice crops destroyed and cities overrun with our military personnel, they\ for help. After the French and Indian War, the Scotch-Irish settlers attacked
\ are doubtless saying secretly of the Vietcong guerillas and of the American forces,\ American Indians. After the attacks on the Conestoga, about 250 Paxton Boys
\ \"A plague on both your houses.\" … Stop the bombing, north and south, end search\ present their grievances to the Pennsylvania legislature. As mentioned in the
\ and destroy offensive sweeps, and confine our military action to holding operations\ information, the Paxton Boys cited resentiment at local elites. The answer is
\ on the ground. Bombing the north has failed to halt or seriously check the flow\ (D).
\ of troops to the south and may, in fact, have prompted a much greater war effort\ - question: "This question refers to the following information.\nOur leaders talk about\
\ by Hanoi.\n—Senator George McGovern, \"The Lessons of Vietnam,\" April 25, 1967\n\ \ stopping aggression from the north, but this was a struggle among groups of\
Which of the following opinions from the 1960s most directly reflects the perspective\ \ Vietnamese until we intervened. We seem bent upon saving the Vietnamese from\
\ of George McGovern's speech?\n(A) Americans must maximize their technological\ \ Ho Chi Minh even if we have to kill them and demolish their country to do\
\ edge in Vietnam. (B) American bombing in Vietnam is step by step leading to progress\ \ it. As the native people survey bombed-out villages, women and children burned\
\ in the war. (C) American bombing in Vietnam is a failure. (D) America must not\ \ by napalm, rice crops destroyed and cities overrun with our military personnel,\
\ give in to defeatism about the war in Vietnam.\nA: Let's think step by step. We\ \ they are doubtless saying secretly of the Vietcong guerillas and of the American\
\ refer to Wikipedia articles on us history for help. \"Stop the bombing\" and \"\ \ forces, \"A plague on both your houses.\" \u2026 Stop the bombing, north and\
Bombing the north has failed to halt or seriously check the flow of troops to the\ \ south, end search and destroy offensive sweeps, and confine our military action\
\ south\" indicate that the perspective of George McGovern's speech is that Amerian\ \ to holding operations on the ground. Bombing the north has failed to halt\
\ bombing in Vietnam is a failure. The answer is (C).\n\nQ: This question refers\ \ or seriously check the flow of troops to the south and may, in fact, have\
\ to the following information.\n\"In the new Code of Laws which I suppose it will\ \ prompted a much greater war effort by Hanoi.\n\u2014Senator George McGovern,\
\ be necessary for you to make I desire you would Remember the Ladies, and be more\ \ \"The Lessons of Vietnam,\" April 25, 1967\nWhich of the following opinions\
\ generous and favorable to them than your ancestors. Do not put such unlimited\ \ from the 1960s most directly reflects the perspective of George McGovern's\
\ power into the hands of the Husbands. Remember all Men would be tyrants if they\ \ speech?\n(A) Americans must maximize their technological edge in Vietnam.\
\ could. If particular care and attention is not paid to the Ladies we are determined\ \ (B) American bombing in Vietnam is step by step leading to progress in the\
\ to foment a Rebellion, and will not hold ourselves bound by any Laws in which\ \ war. (C) American bombing in Vietnam is a failure. (D) America must not give\
\ we have no voice, or Representation.\"\nAbigail Adams, in a letter to John Adams,\ \ in to defeatism about the war in Vietnam."
\ 1776\n\"Special legislation for woman has placed us in a most anomalous position.\ target: Let's think step by step. We refer to Wikipedia articles on us history
\ Women invested with the rights of citizens in one section—voters, jurors, office-holders—crossing\ for help. "Stop the bombing" and "Bombing the north has failed to halt or seriously
\ an imaginary line, are subjects in the next. In some States, a married woman may\ check the flow of troops to the south" indicate that the perspective of George
\ hold property and transact business in her own name; in others, her earnings belong\ McGovern's speech is that Amerian bombing in Vietnam is a failure. The answer
\ to her husband. In some States, a woman may testify against her husband, sue and\ is (C).
\ be sued in the courts; in others, she has no redress in case of damage to person,\ - question: "This question refers to the following information.\n\"In the new Code\
\ property, or character. In case of divorce on account of adultery in the husband,\ \ of Laws which I suppose it will be necessary for you to make I desire you\
\ the innocent wife is held to possess no right to children or property, unless\ \ would Remember the Ladies, and be more generous and favorable to them than\
\ by special decree of the court. But in no State of the Union has the wife the\ \ your ancestors. Do not put such unlimited power into the hands of the Husbands.\
\ right to her own person, or to any part of the joint earnings of the co-partnership\ \ Remember all Men would be tyrants if they could. If particular care and attention\
\ during the life of her husband. In some States women may enter the law schools\ \ is not paid to the Ladies we are determined to foment a Rebellion, and will\
\ and practice in the courts; in others they are forbidden. In some universities\ \ not hold ourselves bound by any Laws in which we have no voice, or Representation.\"\
\ girls enjoy equal educational advantages with boys, while many of the proudest\ \nAbigail Adams, in a letter to John Adams, 1776\n\"Special legislation for\
\ institutions in the land deny them admittance, though the sons of China, Japan\ \ woman has placed us in a most anomalous position. Women invested with the\
\ and Africa are welcomed there. But the privileges already granted in the several\ \ rights of citizens in one section\u2014voters, jurors, office-holders\u2014\
\ States are by no means secure.\"\nSusan B. Anthony, \"Declaration of Rights for\ crossing an imaginary line, are subjects in the next. In some States, a married\
\ Women,\" July 4, 1876\nThe sentiments expressed in the second excerpt by Susan\ \ woman may hold property and transact business in her own name; in others,\
\ B. Anthony are most likely in support of\n(A) the Equal Rights Amendment (B) universal\ \ her earnings belong to her husband. In some States, a woman may testify against\
\ suffrage (C) states' rights (D) prohibition\nA: Let's think step by step. We refer\ \ her husband, sue and be sued in the courts; in others, she has no redress\
\ to Wikipedia articles on us history for help. The above information mentioned\ \ in case of damage to person, property, or character. In case of divorce on\
\ that women are in an anomalous position in terms of legislation. Women's earnings\ \ account of adultery in the husband, the innocent wife is held to possess no\
\ do not belong to themselves, or they cannot testify against her husbands. Susan\ \ right to children or property, unless by special decree of the court. But\
\ believes women should have equal legal rights as men. The answer is (B).\n\nQ:\ \ in no State of the Union has the wife the right to her own person, or to any\
\ This question refers to the following information.\n\"Society in every state is\ \ part of the joint earnings of the co-partnership during the life of her husband.\
\ a blessing, but government even in its best state is but a necessary evil; in\ \ In some States women may enter the law schools and practice in the courts;\
\ its worst state an intolerable one; for when we suffer, or are exposed to the\ \ in others they are forbidden. In some universities girls enjoy equal educational\
\ same miseries by a government, which we might expect in a country without government,\ \ advantages with boys, while many of the proudest institutions in the land\
\ our calamity is heightened by reflecting that we furnish the means by which we\ \ deny them admittance, though the sons of China, Japan and Africa are welcomed\
\ suffer. Government, like dress, is the badge of lost innocence; the palaces of\ \ there. But the privileges already granted in the several States are by no\
\ kings are built on the ruins of the bowers of paradise. For were the impulses\ \ means secure.\"\nSusan B. Anthony, \"Declaration of Rights for Women,\" July\
\ of conscience clear, uniform, and irresistibly obeyed, man would need no other\ \ 4, 1876\nThe sentiments expressed in the second excerpt by Susan B. Anthony\
\ lawgiver; but that not being the case, he finds it necessary to surrender up a\ \ are most likely in support of\n(A) the Equal Rights Amendment (B) universal\
\ part of his property to furnish means for the protection of the rest; and this\ \ suffrage (C) states' rights (D) prohibition"
\ he is induced to do by the same prudence which in every other case advises him\ target: Let's think step by step. We refer to Wikipedia articles on us history
\ out of two evils to choose the least. Wherefore, security being the true design\ for help. The above information mentioned that women are in an anomalous position
\ and end of government, it unanswerably follows that whatever form thereof appears\ in terms of legislation. Women's earnings do not belong to themselves, or they
\ most likely to ensure it to us, with the least expense and greatest benefit, is\ cannot testify against her husbands. Susan believes women should have equal
\ preferable to all others.\"\nThomas Paine, Common Sense, 1776\nWhich of the following\ legal rights as men. The answer is (B).
\ \"miseries\" alluded to above were most condemned by Anti-Federalists of the post-Revolutionary\ - question: 'This question refers to the following information.
\ era?\n(A) Organized response to Bacon's Rebellion (B) Federal response to Shays's\
\ Rebellion (C) Federal response to the Whiskey Rebellion (D) Federal response to\ "Society in every state is a blessing, but government even in its best state
\ Pontiac's Rebellion\nA: Let's think step by step. We refer to Wikipedia articles\ is but a necessary evil; in its worst state an intolerable one; for when we
\ on us history for help. Anti-Federalists do not believe centralized government\ suffer, or are exposed to the same miseries by a government, which we might
\ power, and suspect Washington's military response to Whiskey Rebellion. Bacon's\ expect in a country without government, our calamity is heightened by reflecting
\ Rebellion and Pontiac's Rebellion happen before the Revolution and they can be\ that we furnish the means by which we suffer. Government, like dress, is the
\ ruled out. The answer is (C).\n\n" badge of lost innocence; the palaces of kings are built on the ruins of the
"group": "mmlu_flan_cot_fewshot_humanities" bowers of paradise. For were the impulses of conscience clear, uniform, and
"include": "_mmlu_flan_cot_fewshot_template_yaml" irresistibly obeyed, man would need no other lawgiver; but that not being the
"task": "mmlu_flan_cot_fewshot_high_school_us_history" case, he finds it necessary to surrender up a part of his property to furnish
means for the protection of the rest; and this he is induced to do by the same
prudence which in every other case advises him out of two evils to choose the
least. Wherefore, security being the true design and end of government, it unanswerably
follows that whatever form thereof appears most likely to ensure it to us, with
the least expense and greatest benefit, is preferable to all others."
Thomas Paine, Common Sense, 1776
Which of the following "miseries" alluded to above were most condemned by Anti-Federalists
of the post-Revolutionary era?
(A) Organized response to Bacon''s Rebellion (B) Federal response to Shays''s
Rebellion (C) Federal response to the Whiskey Rebellion (D) Federal response
to Pontiac''s Rebellion'
target: 'Let''s think step by step. We refer to Wikipedia articles on us history
for help. Anti-Federalists do not believe centralized government power, and
suspect Washington''s military response to Whiskey Rebellion. Bacon''s Rebellion
and Pontiac''s Rebellion happen before the Revolution and they can be ruled
out. The answer is (C).'
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_us_history
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml
View file @ a2af2101
"dataset_name": "high_school_world_history" dataset_name: high_school_world_history
"description": "The following are multiple choice questions (with answers) about high\ description: The following are multiple choice questions (with answers) about high
\ school world history.\n\nQ: This question refers to the following information.\n\ school world history.
\"At least one of the [world's] societies would have to somehow enormously increase\ fewshot_config:
\ its productivity [in order to achieve global hegemony]. That quantum jump would\ sampler: first_n
\ have to be made before the various scientific, technological, agricultural, and\ samples:
\ industrial revolutions on which our post-quantum-leap world rests. It could only\ - question: "This question refers to the following information.\n\"At least one of\
\ be accomplished by exploiting the ecosystems, mineral resources, and human assets\ \ the [world's] societies would have to somehow enormously increase its productivity\
\ of whole continents outside the lands of the society making the jump. Western\ \ [in order to achieve global hegemony]. That quantum jump would have to be\
\ Europe did just that by means of its brutality and guns and, more important, by\ \ made before the various scientific, technological, agricultural, and industrial\
\ geographical and ecological luck.\"\nCopyright © 2015 Cambridge University Press.\n\ \ revolutions on which our post-quantum-leap world rests. It could only be accomplished\
Alfred Crosby, historian, Ecological Imperialism, 2004\nThe \"quantum jump\" mentioned\ \ by exploiting the ecosystems, mineral resources, and human assets of whole\
\ in the passage most directly contributed to which of the following developments\ \ continents outside the lands of the society making the jump. Western Europe\
\ in the period 1450–1750 C.E.?\n(A) A breakdown in trade routes through the collapse\ \ did just that by means of its brutality and guns and, more important, by geographical\
\ of the established state structure (B) An increase in the population of the world\ \ and ecological luck.\"\nCopyright \xA9 2015 Cambridge University Press.\n\
\ through more plentiful supplies of food (C) The spread of Chinese and Indian belief\ Alfred Crosby, historian, Ecological Imperialism, 2004\nThe \"quantum jump\"\
\ systems across the world (D) An increase in social unrest\nA: Let's think step\ \ mentioned in the passage most directly contributed to which of the following\
\ by step. We refer to Wikipedia articles on world history for help. The \"quantum\ \ developments in the period 1450\u20131750 C.E.?\n(A) A breakdown in trade\
\ jump\" mentioned in the passage refers to the conquest of the New World and the\ \ routes through the collapse of the established state structure (B) An increase\
\ Columbian Exchange. Choice (A) and (C) did not happen in history. Choice (C) refers\ \ in the population of the world through more plentiful supplies of food (C)\
\ to the human assets. The answer is (B).\n\nQ: This question refers to the following\ \ The spread of Chinese and Indian belief systems across the world (D) An increase\
\ information.\n\"The struggle against neo-colonialism is not aimed at excluding\ \ in social unrest"
\ the capital of the developed world from operating in less developed countries.\ target: Let's think step by step. We refer to Wikipedia articles on world history
\ It is aimed at preventing the financial power of the developed countries being\ for help. The "quantum jump" mentioned in the passage refers to the conquest
\ used in such a way as to impoverish the less developed.\nNon-alignment, as practiced\ of the New World and the Columbian Exchange. Choice (A) and (C) did not happen
\ by Ghana and many other countries, is based on co-operation with all States whether\ in history. Choice (C) refers to the human assets. The answer is (B).
\ they be capitalist, socialist or have a mixed economy. Such a policy, therefore,\ - question: "This question refers to the following information.\n\"The struggle against\
\ involves foreign investment from capitalist countries, but it must be invested\ \ neo-colonialism is not aimed at excluding the capital of the developed world\
\ in accordance with a national plan drawn up by the government of the non-aligned\ \ from operating in less developed countries. It is aimed at preventing the\
\ State with its own interests in mind. The issue is not what return the foreign\ \ financial power of the developed countries being used in such a way as to\
\ investor receives on his investments…The question is one of power. A State in\ \ impoverish the less developed.\nNon-alignment, as practiced by Ghana and many\
\ the grip of neo-colonialism is not master of its own destiny.\"\nKwame Nkrumah,\ \ other countries, is based on co-operation with all States whether they be\
\ Neo-Colonialism, 1965\nWhich of the following provides the best context for Nkrumah's\ \ capitalist, socialist or have a mixed economy. Such a policy, therefore, involves\
\ writings?\n(A) The Industrial Revolution (B) Decolonization (C) Regional Free\ \ foreign investment from capitalist countries, but it must be invested in accordance\
\ Trade Associations (D) Autarky\nA: Let's think step by step. We refer to Wikipedia\ \ with a national plan drawn up by the government of the non-aligned State with\
\ articles on world history for help. The passage expresses a point that the successful\ \ its own interests in mind. The issue is not what return the foreign investor\
\ fight against neo-colonialism were in danger and the newly independent nations\ \ receives on his investments\u2026The question is one of power. A State in\
\ like Ghana may be re-colonized via financial power of the developed countries.\ \ the grip of neo-colonialism is not master of its own destiny.\"\nKwame Nkrumah,\
\ The answer is (B).\n\nQ: This question refers to the following information.\n\"\ \ Neo-Colonialism, 1965\nWhich of the following provides the best context for\
Indeed, as both the fatwas of distinguished [scholars] who base their opinion on\ \ Nkrumah's writings?\n(A) The Industrial Revolution (B) Decolonization (C)\
\ reason and tradition alike and the consensus of the Sunni community agree that\ \ Regional Free Trade Associations (D) Autarky"
\ the ancient obligation of extirpation, extermination, and expulsion of evil innovation\ target: Let's think step by step. We refer to Wikipedia articles on world history
\ must be the aim of our exalted aspiration, for \"Religious zeal is a victory for\ for help. The passage expresses a point that the successful fight against neo-colonialism
\ the Faith of God the Beneficent\"; then, in accordance with the words of the Prophet\ were in danger and the newly independent nations like Ghana may be re-colonized
\ (Peace upon him!) \"Whosoever introduces evil innovation into our order must be\ via financial power of the developed countries. The answer is (B).
\ expelled\" and \"Whosoever does aught against our order must be expelled,\" action\ - question: "This question refers to the following information.\n\"Indeed, as both\
\ has become necessary and exigent…\"\nLetter from Ottoman Sultan Selim I to Safavid\ \ the fatwas of distinguished [scholars] who base their opinion on reason and\
\ Shah Ismail I, 1514\nThe letter from Selim I is most clearly an example of which\ \ tradition alike and the consensus of the Sunni community agree that the ancient\
\ of the following?\n(A) The maintenance of military supremacy at all costs (B)\ \ obligation of extirpation, extermination, and expulsion of evil innovation\
\ Expanding tensions between religious sects (C) Factors that brought about the\ \ must be the aim of our exalted aspiration, for \"Religious zeal is a victory\
\ collapse of the Ottoman Empire (D) Peacemaking efforts among the Islamic empires\n\ \ for the Faith of God the Beneficent\"; then, in accordance with the words\
A: Let's think step by step. We refer to Wikipedia articles on world history for\ \ of the Prophet (Peace upon him!) \"Whosoever introduces evil innovation into\
\ help. The passage is an example of expanding tensions between Selim and Ismail.\ \ our order must be expelled\" and \"Whosoever does aught against our order\
\ In the passage the Selim references the fatwa and the consensus of the Sunni community\ \ must be expelled,\" action has become necessary and exigent\u2026\"\nLetter\
\ to against whosoever introduces evil. The answer is (B).\n\nQ: This question refers\ \ from Ottoman Sultan Selim I to Safavid Shah Ismail I, 1514\nThe letter from\
\ to the following information.\n\"The real grievance of the worker is the insecurity\ \ Selim I is most clearly an example of which of the following?\n(A) The maintenance\
\ of his existence; he is not sure that he will always have work, he is not sure\ \ of military supremacy at all costs (B) Expanding tensions between religious\
\ that he will always be healthy, and he foresees that he will one day be old and\ \ sects (C) Factors that brought about the collapse of the Ottoman Empire (D)\
\ unfit to work. If he falls into poverty, even if only through a prolonged illness,\ \ Peacemaking efforts among the Islamic empires"
\ he is then completely helpless, exam_ins to his own devices, and society does\ target: Let's think step by step. We refer to Wikipedia articles on world history
\ not currently recognize any real obligation towards him beyond the usual help\ for help. The passage is an example of expanding tensions between Selim and
\ for the poor, even if he has been working all the time ever so faithfully and\ Ismail. In the passage the Selim references the fatwa and the consensus of the
\ diligently. The usual help for the poor, however, leaves a lot to be desired,\ Sunni community to against whosoever introduces evil. The answer is (B).
\ especially in large cities, where it is very much worse than in the country.\"\ - question: 'This question refers to the following information.
\nOtto von Bismarck, 1884\nOtto von Bismarck likely made this speech in reaction\
\ to which of the following issues?\n(A) Social acceptance of child labor (B) Declining\ "The real grievance of the worker is the insecurity of his existence; he is
\ life expectancy in Germany (C) Criticisms of German trade tariffs (D) Negative\ not sure that he will always have work, he is not sure that he will always be
\ effects attributed to industrial capitalism\nA: Let's think step by step. We refer\ healthy, and he foresees that he will one day be old and unfit to work. If he
\ to Wikipedia articles on world history for help. The passage talks about the grievance\ falls into poverty, even if only through a prolonged illness, he is then completely
\ of the work under the industrial capitalism. The answer is (D).\n\nQ: This question\ helpless, exam_ins to his own devices, and society does not currently recognize
\ refers to the following information.\nHe contains all works and desires and all\ any real obligation towards him beyond the usual help for the poor, even if
\ perfumes and all tastes. He enfolds the whole universe and in silence is loving\ he has been working all the time ever so faithfully and diligently. The usual
\ to all. This is the Spirit that is in my heart, this is Brahman. To him I shall\ help for the poor, however, leaves a lot to be desired, especially in large
\ come when I go beyond this life, and to him will come he who has faith and doubts\ cities, where it is very much worse than in the country."
\ not.\n—The Upanishads, India, c. 1000 BCE\nTo which religion does the speaker\
\ most likely belong?\n(A) Hinduism (B) Buddhism (C) Shintoism (D) Zoroastrianism\n\ Otto von Bismarck, 1884
A: Let's think step by step. We refer to Wikipedia articles on world history for\
\ help. Brahman refers to the ultimate reality of all things in the Hindu religion.\ Otto von Bismarck likely made this speech in reaction to which of the following
\ In contrast, Buddhism does not have a concept of supreme God. The answer is (A).\n\n" issues?
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml" (A) Social acceptance of child labor (B) Declining life expectancy in Germany
"task": "mmlu_flan_cot_fewshot_high_school_world_history" (C) Criticisms of German trade tariffs (D) Negative effects attributed to industrial
capitalism'
target: Let's think step by step. We refer to Wikipedia articles on world history
for help. The passage talks about the grievance of the work under the industrial
capitalism. The answer is (D).
- question: "This question refers to the following information.\nHe contains all works\
\ and desires and all perfumes and all tastes. He enfolds the whole universe\
\ and in silence is loving to all. This is the Spirit that is in my heart, this\
\ is Brahman. To him I shall come when I go beyond this life, and to him will\
\ come he who has faith and doubts not.\n\u2014The Upanishads, India, c. 1000\
\ BCE\nTo which religion does the speaker most likely belong?\n(A) Hinduism\
\ (B) Buddhism (C) Shintoism (D) Zoroastrianism"
target: 'Let''s think step by step. We refer to Wikipedia articles on world history
for help. Brahman refers to the ultimate reality of all things in the Hindu
religion. In contrast, Buddhism does not have a concept of supreme God. The
answer is (A).'
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_world_history
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml
View file @ a2af2101
"dataset_name": "human_aging" dataset_name: human_aging
"description": "The following are multiple choice questions (with answers) about human\ description: The following are multiple choice questions (with answers) about human
\ aging.\n\nQ: All other things being equal, which of the following persons is more\ aging.
\ likely to show osteoporosis?\n(A) An older Hispanic American woman (B) An older\ fewshot_config:
\ African American woman (C) An older Asian American woman (D) An older Native American\ sampler: first_n
\ woman\nA: Let's think step by step. We refer to Wikipedia articles on human aging\ samples:
\ for help. Although osteoporosis can occur at any age, the risk is higher for older\ - question: 'All other things being equal, which of the following persons is more likely
\ people. It is most common in Asian and non-Hispanic white women. The answer is\ to show osteoporosis?
\ (C).\n\nQ: The finding that adults tend to remember events from their adolescence\
\ better than from other periods in their lives is referred to as the\n(A) Adolescence\ (A) An older Hispanic American woman (B) An older African American woman (C)
\ advantage (B) Reminiscence bump (C) Memorial memorial (D) Quadratic retrieval\ An older Asian American woman (D) An older Native American woman'
\ spike\nA: Let's think step by step. We refer to Wikipedia articles on human aging\ target: Let's think step by step. We refer to Wikipedia articles on human aging
\ for help. Reminiscence bump is a phenomenon that older adults tend to recollect\ for help. Although osteoporosis can occur at any age, the risk is higher for
\ events during their young ages. People usually have a period of childhood amnesia\ older people. It is most common in Asian and non-Hispanic white women. The answer
\ from birth to around age 5, and a reminiscence bump between 10 and 30. The answer\ is (C).
\ is (B).\n\nQ: Which element in tobacco smoke is responsible for cancers?\n(A)\ - question: 'The finding that adults tend to remember events from their adolescence
\ Nicotine (B) Tar (C) Carbon monoxide (D) Smoke particles\nA: Let's think step\ better than from other periods in their lives is referred to as the
\ by step. We refer to Wikipedia articles on human aging for help. The benzene,\
\ acrylamide and acrylonitrile in tar interact with the lungs and cause DNA mutations\ (A) Adolescence advantage (B) Reminiscence bump (C) Memorial memorial (D) Quadratic
\ in cells of the lungs, and lead to cancer. The answer is (B).\n\nQ: When older\ retrieval spike'
\ adults move to a new state after retirement, which of the following is the more\ target: Let's think step by step. We refer to Wikipedia articles on human aging
\ likely destination?\n(A) Texas (B) California (C) Hawaii (D) Vermont\nA: Let's\ for help. Reminiscence bump is a phenomenon that older adults tend to recollect
\ think step by step. We refer to Wikipedia articles on human aging for help. Texas\ events during their young ages. People usually have a period of childhood amnesia
\ does not have state tax, and has low cost of living compared with the other three\ from birth to around age 5, and a reminiscence bump between 10 and 30. The answer
\ options. The answer is (A).\n\n" is (B).
"group": "mmlu_flan_cot_fewshot_other" - question: 'Which element in tobacco smoke is responsible for cancers?
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_human_aging" (A) Nicotine (B) Tar (C) Carbon monoxide (D) Smoke particles'
target: Let's think step by step. We refer to Wikipedia articles on human aging
for help. The benzene, acrylamide and acrylonitrile in tar interact with the
lungs and cause DNA mutations in cells of the lungs, and lead to cancer. The
answer is (B).
- question: 'When older adults move to a new state after retirement, which of the following
is the more likely destination?
(A) Texas (B) California (C) Hawaii (D) Vermont'
target: 'Let''s think step by step. We refer to Wikipedia articles on human aging
for help. Texas does not have state tax, and has low cost of living compared
with the other three options. The answer is (A).'
tag: mmlu_flan_cot_fewshot_other
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_human_aging
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml
View file @ a2af2101
"dataset_name": "human_sexuality" dataset_name: human_sexuality
"description": "The following are multiple choice questions (with answers) about human\ description: The following are multiple choice questions (with answers) about human
\ sexuality.\n\nQ: The most common disorder among men who seek sexual therapy is:\n\ sexuality.
(A) premature ejaculation (B) inhibited ejaculation (C) erectile disorder (D) ejaculatory\ fewshot_config:
\ disorder\nA: Let's think step by step. We refer to Wikipedia articles on human\ sampler: first_n
\ sexuality for help. The most common type of arousal disorder for men is erectile\ samples:
\ dysfunction, meaning that a person is not able to get any physical satisfaction\ - question: 'The most common disorder among men who seek sexual therapy is:
\ from sexual activity although he may be interested in it. The answer is (C).\n\
\nQ: A woman who knows she has active herpes and untreated syphilis but continues\ (A) premature ejaculation (B) inhibited ejaculation (C) erectile disorder (D)
\ to have sex without informing her partners of her condition has, in psychoanalytic\ ejaculatory disorder'
\ terms:\n(A) a strong ego (B) a weak superego (C) a weak id (D) a strong superego\n\ target: Let's think step by step. We refer to Wikipedia articles on human sexuality
A: Let's think step by step. We refer to Wikipedia articles on human sexuality for\ for help. The most common type of arousal disorder for men is erectile dysfunction,
\ help. A person with weak superego tends to be delinquent, criminal or have antisocial\ meaning that a person is not able to get any physical satisfaction from sexual
\ personality. The action of the woman who knows she has active venereal disease\ activity although he may be interested in it. The answer is (C).
\ but still have sex with her partners indicate she may has antisocial personality.\ - question: 'A woman who knows she has active herpes and untreated syphilis but continues
\ The answer is (B).\n\nQ: The nature of homosexual activities that occur during\ to have sex without informing her partners of her condition has, in psychoanalytic
\ preadolescence include all but which of the following?\n(A) sexual intercourse\ terms:
\ (B) circle jerks (C) exhibitionism (D) touching each other's genitals\nA: Let's\
\ think step by step. We refer to Wikipedia articles on human sexuality for help.\ (A) a strong ego (B) a weak superego (C) a weak id (D) a strong superego'
\ Sexual intercourse is prohibited during preadolescence. The answer is (A).\n\n\ target: Let's think step by step. We refer to Wikipedia articles on human sexuality
Q: Women's ability to have multiple orgasms is primarily due to:\n(A) the fact that\ for help. A person with weak superego tends to be delinquent, criminal or have
\ they do not have a refractory period. (B) the response of the inner layers of\ antisocial personality. The action of the woman who knows she has active venereal
\ the vagina. (C) having alternating orgasms in different locations. (D) the G-Spot.\n\ disease but still have sex with her partners indicate she may has antisocial
A: Let's think step by step. We refer to Wikipedia articles on human sexuality for\ personality. The answer is (B).
\ help. The refractory period is the time when a person is not able to be erect\ - question: 'The nature of homosexual activities that occur during preadolescence include
\ or is not interested in sex. The answer is (A).\n\nQ: Morning sickness is typically\ all but which of the following?
\ a problem:\n(A) during the first trimester (B) during the second trimester (C)\
\ during the third trimester (D) all through the pregnancy\nA: Let's think step\ (A) sexual intercourse (B) circle jerks (C) exhibitionism (D) touching each
\ by step. We refer to Wikipedia articles on human sexuality for help. Morning sickness\ other''s genitals'
\ usually begins by nine weeks after conception, corresponding to the first trimester.\ target: Let's think step by step. We refer to Wikipedia articles on human sexuality
\ The answer is (A).\n\n" for help. Sexual intercourse is prohibited during preadolescence. The answer
"group": "mmlu_flan_cot_fewshot_social_sciences" is (A).
"include": "_mmlu_flan_cot_fewshot_template_yaml" - question: 'Women''s ability to have multiple orgasms is primarily due to:
"task": "mmlu_flan_cot_fewshot_human_sexuality"
(A) the fact that they do not have a refractory period. (B) the response of
the inner layers of the vagina. (C) having alternating orgasms in different
locations. (D) the G-Spot.'
target: Let's think step by step. We refer to Wikipedia articles on human sexuality
for help. The refractory period is the time when a person is not able to be
erect or is not interested in sex. The answer is (A).
- question: 'Morning sickness is typically a problem:
(A) during the first trimester (B) during the second trimester (C) during the
third trimester (D) all through the pregnancy'
target: 'Let''s think step by step. We refer to Wikipedia articles on human sexuality
for help. Morning sickness usually begins by nine weeks after conception, corresponding
to the first trimester. The answer is (A).'
tag: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_human_sexuality
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml
View file @ a2af2101
"dataset_name": "international_law" dataset_name: international_law
"description": "The following are multiple choice questions (with answers) about international\ description: The following are multiple choice questions (with answers) about international
\ law.\n\nQ: How the consent to be bound of a State may be expressed?\n(A) The consent\ law.
\ of a State to be bound is expressed only by ratification (B) The consent of a\ fewshot_config:
\ state to be bound by a treaty may be expressed by signature, ratification, acceptance,\ sampler: first_n
\ approval or accession (C) The consent of a State to be bound is expressed by signature\ samples:
\ (D) The consent of a State to be bound is expressed by whatever means they choose\n\ - question: 'How the consent to be bound of a State may be expressed?
A: Let's think step by step. We refer to Wikipedia articles on international law\
\ for help. Article 11 of Vienna Convention on the Law of Treaties signed in 1969\ (A) The consent of a State to be bound is expressed only by ratification (B)
\ states that \"the consent of a State to be bound by a treaty may be expressed\ The consent of a state to be bound by a treaty may be expressed by signature,
\ by signature, exchange of instruments constituting a treaty, ratification, acceptance,\ ratification, acceptance, approval or accession (C) The consent of a State to
\ approval or accession, or by any other means if so agreed.\" (B) is the most precise\ be bound is expressed by signature (D) The consent of a State to be bound is
\ and accurate answer. The answer is (B).\n\nQ: What is the judge ad hoc?\n(A) If\ expressed by whatever means they choose'
\ a party to a contentious case before the ICJ does not have a national sitting\ target: Let's think step by step. We refer to Wikipedia articles on international
\ as judge, it is entitled to nominate someone as a judge solely for that case,\ law for help. Article 11 of Vienna Convention on the Law of Treaties signed
\ with the title of judge ad hoc (B) Judge ad hoc is the member of the bench of\ in 1969 states that "the consent of a State to be bound by a treaty may be expressed
\ the ICJ with a casting vote (C) Judge ad hoc is a surrogate judge, in case a judge\ by signature, exchange of instruments constituting a treaty, ratification, acceptance,
\ is disqualified or passes away (D) Judge ad hoc is the judge that each party will\ approval or accession, or by any other means if so agreed." (B) is the most
\ always nominate in every contentious case\nA: Let's think step by step. We refer\ precise and accurate answer. The answer is (B).
\ to Wikipedia articles on international law for help. As \"ad hoc\" implies, a\ - question: 'What is the judge ad hoc?
\ judge ad hoc is appointed only for a specific case or period, when a party to\
\ a contentious case before the International Court of Justice does not have a regular\ (A) If a party to a contentious case before the ICJ does not have a national
\ national sitting as judge. The answer is (A).\n\nQ: When 'consent' can serve as\ sitting as judge, it is entitled to nominate someone as a judge solely for that
\ a circumstance precluding the wrongfulness of a State conduct?\n(A) Consent can\ case, with the title of judge ad hoc (B) Judge ad hoc is the member of the bench
\ serve as a circumstance precluding the wrongfulness whenever it is given (B) Consent\ of the ICJ with a casting vote (C) Judge ad hoc is a surrogate judge, in case
\ can never serve as a circumstance precluding wrongfulness (C) Consent can serve\ a judge is disqualified or passes away (D) Judge ad hoc is the judge that each
\ as a circumstance precluding wrongfulness, provided the consent is valid and to\ party will always nominate in every contentious case'
\ the extent that the conduct remains within the limits of the consent given (D)\ target: Let's think step by step. We refer to Wikipedia articles on international
\ Consent can always serve as a circumstance precluding wrongfulness, no matter\ law for help. As "ad hoc" implies, a judge ad hoc is appointed only for a specific
\ which organ of the State gives it\nA: Let's think step by step. We refer to Wikipedia\ case or period, when a party to a contentious case before the International
\ articles on international law for help. Valid consent can serve as a circumstance\ Court of Justice does not have a regular national sitting as judge. The answer
\ precluding the wrongfulness of a State conduct if the conduct remains within the\ is (A).
\ limits of that consent, according to Chapter V of the Responsibility of States\ - question: 'When ''consent'' can serve as a circumstance precluding the wrongfulness
\ for Internationally Wrongful Acts, 2001, United Nations. The answer is (C).\n\n\ of a State conduct?
Q: Would a reservation to the definition of torture in the ICCPR be acceptable in\
\ contemporary practice?\n(A) This is an acceptable reservation if the reserving\ (A) Consent can serve as a circumstance precluding the wrongfulness whenever
\ country's legislation employs a different definition (B) This is an unacceptable\ it is given (B) Consent can never serve as a circumstance precluding wrongfulness
\ reservation because it contravenes the object and purpose of the ICCPR (C) This\ (C) Consent can serve as a circumstance precluding wrongfulness, provided the
\ is an unacceptable reservation because the definition of torture in the ICCPR\ consent is valid and to the extent that the conduct remains within the limits
\ is consistent with customary international law (D) This is an acceptable reservation\ of the consent given (D) Consent can always serve as a circumstance precluding
\ because under general international law States have the right to enter reservations\ wrongfulness, no matter which organ of the State gives it'
\ to treaties\nA: Let's think step by step. We refer to Wikipedia articles on international\ target: Let's think step by step. We refer to Wikipedia articles on international
\ law for help. For it contravenes the object and purpose of the ICCPR, this is\ law for help. Valid consent can serve as a circumstance precluding the wrongfulness
\ an unacceptable reservation in contemporary practice. The answer is (B).\n\nQ:\ of a State conduct if the conduct remains within the limits of that consent,
\ What types of force does Article 2(4) of the UN Charter prohibit?\n(A) Article\ according to Chapter V of the Responsibility of States for Internationally Wrongful
\ 2(4) encompasses only armed force (B) Article 2(4) encompasses all types of force,\ Acts, 2001, United Nations. The answer is (C).
\ including sanctions (C) Article 2(4) encompasses all interference in the domestic\ - question: 'Would a reservation to the definition of torture in the ICCPR be acceptable
\ affairs of States (D) Article 2(4) encompasses force directed only against a State's\ in contemporary practice?
\ territorial integrity\nA: Let's think step by step. We refer to Wikipedia articles\
\ on international law for help. Article 2(4) of the UN Charter prohibits states\ (A) This is an acceptable reservation if the reserving country''s legislation
\ from using armed forces in their international relations. The answer is (A).\n\n" employs a different definition (B) This is an unacceptable reservation because
"group": "mmlu_flan_cot_fewshot_humanities" it contravenes the object and purpose of the ICCPR (C) This is an unacceptable
"include": "_mmlu_flan_cot_fewshot_template_yaml" reservation because the definition of torture in the ICCPR is consistent with
"task": "mmlu_flan_cot_fewshot_international_law" customary international law (D) This is an acceptable reservation because under
general international law States have the right to enter reservations to treaties'
target: Let's think step by step. We refer to Wikipedia articles on international
law for help. For it contravenes the object and purpose of the ICCPR, this is
an unacceptable reservation in contemporary practice. The answer is (B).
- question: 'What types of force does Article 2(4) of the UN Charter prohibit?
(A) Article 2(4) encompasses only armed force (B) Article 2(4) encompasses all
types of force, including sanctions (C) Article 2(4) encompasses all interference
in the domestic affairs of States (D) Article 2(4) encompasses force directed
only against a State''s territorial integrity'
target: 'Let''s think step by step. We refer to Wikipedia articles on international
law for help. Article 2(4) of the UN Charter prohibits states from using armed
forces in their international relations. The answer is (A).'
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_international_law
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml
View file @ a2af2101
"dataset_name": "jurisprudence" dataset_name: jurisprudence
"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ description: The following are multiple choice questions (with answers) about jurisprudence.
\nQ: Iverson Jewelers wrote a letter to Miller, 'We have received an exceptionally\ fewshot_config:
\ fine self winding Rolox watch which we will sell to you at a very favorable price.'\n\ sampler: first_n
(A) The letter is an offer to sell (B) A valid offer cannot be made by letter. (C)\ samples:
\ The letter contains a valid offer which will terminate within a reasonable time.\ - question: 'Iverson Jewelers wrote a letter to Miller, ''We have received an exceptionally
\ (D) The letter lacks one of the essential elements of an offer.\nA: Let's think\ fine self winding Rolox watch which we will sell to you at a very favorable
\ step by step. We refer to Wikipedia articles on jurisprudence for help. An offer\ price.''
\ shows the intent to enter into a mutually-beneficial contract with specific terms.\
\ An offer can be made by a letter. While this letter indicates the willingness\ (A) The letter is an offer to sell (B) A valid offer cannot be made by letter.
\ to sell, the lack of specific terms, such as transaction price and offer expiration\ (C) The letter contains a valid offer which will terminate within a reasonable
\ date, makes it an incomplete offer. The answer is (D).\n\nQ: Functions of the\ time. (D) The letter lacks one of the essential elements of an offer.'
\ law include all but which of the following?\n(A) maximizing individual freedom\ target: Let's think step by step. We refer to Wikipedia articles on jurisprudence
\ (B) providing a basis for compromise (C) keeping the peace (D) promoting the principles\ for help. An offer shows the intent to enter into a mutually-beneficial contract
\ of the free enterprise system\nA: Let's think step by step. We refer to Wikipedia\ with specific terms. An offer can be made by a letter. While this letter indicates
\ articles on jurisprudence for help. Laws are fundamentally about helping resolve\ the willingness to sell, the lack of specific terms, such as transaction price
\ disputes between individuals, and therefore essential for maximizing individual\ and offer expiration date, makes it an incomplete offer. The answer is (D).
\ freedom, providing a basis for compromise, and keeping the peace. The answer is\ - question: 'Functions of the law include all but which of the following?
\ (D).\n\nQ: The ________ School of jurisprudence postulates that the law is based\
\ on what is \"correct.\"\n(A) Natural Law (B) Analytical (C) Historical (D) Sociological\n\ (A) maximizing individual freedom (B) providing a basis for compromise (C) keeping
A: Let's think step by step. We refer to Wikipedia articles on jurisprudence for\ the peace (D) promoting the principles of the free enterprise system'
\ help. Natural Law School of jurisprudence focuses on the laws of nature, and states\ target: Let's think step by step. We refer to Wikipedia articles on jurisprudence
\ that the law should be based on ethics, morals, and what is \"correct\". Analytical\ for help. Laws are fundamentally about helping resolve disputes between individuals,
\ deals with the law as it already exists, Historical postulates that the law was\ and therefore essential for maximizing individual freedom, providing a basis
\ found and not made, and Sociological studies how the law and society impact each\ for compromise, and keeping the peace. The answer is (D).
\ other. The answer is (A).\n\nQ: Which word best summarizes Weber's explanation\ - question: 'The ________ School of jurisprudence postulates that the law is based
\ of the development of formally rational law?\n(A) Authority. (B) Charisma. (C)\ on what is "correct."
\ Co-operation. (D) Capitalism.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on jurisprudence for help. Weber explained the development of formal\ (A) Natural Law (B) Analytical (C) Historical (D) Sociological'
\ rationality in laws as how the modern society moved from tradition to rationality,\ target: Let's think step by step. We refer to Wikipedia articles on jurisprudence
\ where people decide actions based less on how they were culturally done and more\ for help. Natural Law School of jurisprudence focuses on the laws of nature,
\ on expected utilities. How rational individuals optimize efficiency of accomplishing\ and states that the law should be based on ethics, morals, and what is "correct".
\ tasks for higher rewards is a core principle of Capitalism. The answer is (D).\n\ Analytical deals with the law as it already exists, Historical postulates that
\nQ: Which position does Rawls claim is the least likely to be adopted by the POP\ the law was found and not made, and Sociological studies how the law and society
\ (people in the original position)?\n(A) The POP would choose equality above liberty.\ impact each other. The answer is (A).
\ (B) The POP would opt for the 'maximin' strategy. (C) The POP would opt for the\ - question: 'Which word best summarizes Weber''s explanation of the development of
\ 'difference principle'. (D) The POP would reject the 'system of natural liberty.'\n\ formally rational law?
A: Let's think step by step. We refer to Wikipedia articles on jurisprudence for\
\ help. The POP would opt for the 'maximin' strategy, opt for the 'difference principle',\ (A) Authority. (B) Charisma. (C) Co-operation. (D) Capitalism.'
\ and reject the 'system of natural liberty', but the POP would not choose equality\ target: Let's think step by step. We refer to Wikipedia articles on jurisprudence
\ above liberty, since the POP assume both equal and free citizens. The answer is\ for help. Weber explained the development of formal rationality in laws as how
\ (A).\n\n" the modern society moved from tradition to rationality, where people decide
"group": "mmlu_flan_cot_fewshot_humanities" actions based less on how they were culturally done and more on expected utilities.
"include": "_mmlu_flan_cot_fewshot_template_yaml" How rational individuals optimize efficiency of accomplishing tasks for higher
"task": "mmlu_flan_cot_fewshot_jurisprudence" rewards is a core principle of Capitalism. The answer is (D).
- question: 'Which position does Rawls claim is the least likely to be adopted by the
POP (people in the original position)?
(A) The POP would choose equality above liberty. (B) The POP would opt for the
''maximin'' strategy. (C) The POP would opt for the ''difference principle''.
(D) The POP would reject the ''system of natural liberty.'''
target: 'Let''s think step by step. We refer to Wikipedia articles on jurisprudence
for help. The POP would opt for the ''maximin'' strategy, opt for the ''difference
principle'', and reject the ''system of natural liberty'', but the POP would
not choose equality above liberty, since the POP assume both equal and free
citizens. The answer is (A).'
tag: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_jurisprudence
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment