Skip to content

GitLab

Commit 6ac42518 authored by lintangsutawika's avatar lintangsutawika
Browse files

Merge branch 'big-refactor' of... 

Merge branch 'big-refactor' of https://github.com/EleutherAI/lm-evaluation-harness into openai_completions
parents 9c3ba7d4 e3644fcc
Hide whitespace changes
Inline Side-by-side
Showing with 1090 additions and 0 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "business_ethics"
"description": "The following are multiple choice questions (with answers) about business\
\ ethics.\n\nQ: In contrast to _______, _______ aim to reward favourable behaviour\
\ by companies. The success of such campaigns have been heightened through the use\
\ of ___________, which allow campaigns to facilitate the company in achieving _________\
\ .\n(A) Buycotts, Boycotts, Blockchain technology, Charitable donations (B) Buycotts,\
\ Boycotts, Digital technology, Increased Sales (C) Boycotts, Buyalls, Blockchain\
\ technology, Charitable donations (D) Boycotts, Buycotts, Digital technology, Increased\
\ Sales\nA: Let's think step by step. We refer to Wikipedia articles on business\
\ ethics for help. The sentence that best uses the possible options above is “In\
\ contrast to *boycotts*, *buycotts* aim to reward favourable behavior by companies.\
\ The success of such campaigns have been heightened through the use of *digital\
\ technology*, which allow campaigns to facilitate the company in achieving *increased\
\ sales*.” The answer is (D).\n\nQ: _______ is the direct attempt to formally or\
\ informally manage ethical issues or problems, through specific policies, practices\
\ and programmes.\n(A) Corporate social responsibility (B) Business ethics management\
\ (C) Sustainability (D) Environmental management\nA: Let's think step by step.\
\ We refer to Wikipedia articles on business ethics for help. The direct attempt\
\ manage ethical issues through specific policies, practices, and programs is business\
\ ethics management. The answer is (B).\n\nQ: Three contrasting tactics that CSO's\
\ can engage in to meet their aims are ________ which typically involves research\
\ and communication, ________, which may involve physically attacking a company's\
\ operations or ________, often involving some form of _______.\n(A) Non-violent\
\ direct action, Violent direct action, Indirect action, Boycott (B) Indirect action,\
\ Instrumental action, Non-violent direct action, Information campaign (C) Indirect\
\ action, Violent direct action, Non-violent direct-action Boycott (D) Non-violent\
\ direct action, Instrumental action, Indirect action, Information campaign\nA:\
\ Let's think step by step. We refer to Wikipedia articles on business ethics for\
\ help. The sentence that best uses the possible options above is “Three contrasting\
\ tactics that CSO's can engage in to meet their aims are *indirect action*, which\
\ typically involves research and communication, *violent direct action*, which\
\ may involve physically attacking a company's operations or *non-violent direct\
\ action*, often involving some form of *boycott*.” The answer is (C).\n\nQ: To\
\ ensure the independence of the non-executive board members, there are a number\
\ of steps which can be taken, which include non-executives being drawn from _______\
\ the company, being appointed for a _________ time period as well as being appointed\
\ _________.\n(A) Outside, Limited, Independently (B) Inside, Limited, Intermittently\
\ (C) Outside, Unlimited, Intermittently (D) Inside, Unlimited, Independently\n\
A: Let's think step by step. We refer to Wikipedia articles on business ethics for\
\ help. The sentence that best uses the possible options above is “To ensure the\
\ independence of the non-executive board members, there are a number of steps which\
\ can be taken, which include non-executives being draw from *outside* the company,\
\ being appointed for a *limited* time period as well as being imported *independently*.\
\ The answer is (A).\n\nQ: Beyond the business case for engaging in CSR there are\
\ a number of moral arguments relating to: negative _______, the _______that corporations\
\ possess and the ________ of business and society.\n(A) Externalities, Power, Independence\
\ (B) Publicity, Insubstantial resources, Mutual dependence (C) Publicity, Power,\
\ Independence (D) Externalities, Power, Mutual dependence\nA: Let's think step\
\ by step. We refer to Wikipedia articles on business ethics for help. The sentence\
\ that best uses the possible options above is “Beyond the business case for engaging\
\ the CSR there are a number of moral arguments relating to: negative *externalities*,\
\ the *power* that corporations possess and the *mutual independence* of business\
\ and society. The answer is (D)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_business_ethics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "clinical_knowledge"
"description": "The following are multiple choice questions (with answers) about clinical\
\ knowledge.\n\nQ: Glycolysis is the name given to the pathway involving the conversion\
\ of:\n(A) glycogen to glucose-1-phosphate. (B) glycogen or glucose to fructose.\
\ (C) glycogen or glucose to pyruvate or lactate. (D) glycogen or glucose to pyruvate\
\ or acetyl CoA.\nA: Let's think step by step. We refer to Wikipedia articles on\
\ clinical knowledge for help. Glycolysis is the name given to the pathway involving\
\ conversion of glycogen or glucose to pyruvate or lactate. The answer is (C).\n\
\nQ: What is the difference between a male and a female catheter?\n(A) Male and\
\ female catheters are different colours. (B) Male catheters are longer than female\
\ catheters. (C) Male catheters are bigger than female catheters. (D) Female catheters\
\ are longer than male catheters.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on clinical knowledge for help. The difference between a male and female\
\ catheter is that male catheters tend to be longer than female catheters. The answer\
\ is (B).\n\nQ: How many attempts should you make to cannulate a patient before\
\ passing the job on to a senior colleague, according to the medical knowledge of\
\ 2020?\n(A) 4 (B) 3 (C) 2 (D) 1\nA: Let's think step by step. We refer to Wikipedia\
\ articles on clinical knowledge for help. According to the medical protocol as\
\ of 2020, you should make two attempts to cannulate a patient before passing the\
\ job on to a more-senior practitioner. The answer is (C).\n\nQ: In the assessment\
\ of the hand function which of the following is true?\n(A) Abduction of the thumb\
\ is supplied by spinal root T2 (B) Opposition of the thumb by opponens policis\
\ is supplied by spinal root T1 (C) Finger adduction is supplied by the median nerve\
\ (D) Finger abduction is mediated by the palmar interossei\nA: Let's think step\
\ by step. We refer to Wikipedia articles on clinical knowledge for help. Of all\
\ the options, it is only true that the opposition of the thumb by opponens pollicis\
\ is supplied by spinal root T1. The answer is (B).\n\nQ: The energy for all forms\
\ of muscle contraction is provided by:\n(A) ATP. (B) ADP. (C) phosphocreatine.\
\ (D) oxidative phosphorylation.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on clinical knowledge for help. The energy for muscular contraction is\
\ provided by ATP (adenosine triphosphate), which is the powerhouse of the cell.\
\ The answer is (A)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_clinical_knowledge"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_biology"
"description": "The following are multiple choice questions (with answers) about college\
\ biology.\n\nQ: Which of the following represents an accurate statement concerning\
\ arthropods?\n(A) They possess an exoskeleton composed primarily of peptidoglycan.\
\ (B) They possess an open circulatory system with a dorsal heart. (C) They are\
\ members of a biologically unsuccessful phylum incapable of exploiting diverse\
\ habitats and nutrition sources. (D) They lack paired, jointed appendages.\nA:\
\ Let's think step by step. Peptidoglycan is known to comprise the plasma membrane\
\ of most bacteria, rather than the exoskeleton of arthropods, which is made of\
\ chitin, which rules out (A). The answer (C) is false because arthropods are a\
\ highly successful phylum. Likewise, arthropods have paired, jointed appendages,\
\ which rules out (D). The only remaining option is (B), as arthropods have an open\
\ circulatory system with a dorsal tubular heart. The answer is (B).\n\nQ: In a\
\ given population, 1 out of every 400 people has a cancer caused by a completely\
\ recessive allele, b. Assuming the population is in Hardy-Weinberg equilibrium,\
\ which of the following is the expected proportion of individuals who carry the\
\ b allele but are not expected to develop the cancer?\n(A) 1/400 (B) 19/400 (C)\
\ 20/400 (D) 38/400\nA: Let's think step by step. According to the Hardy Weinberg\
\ Law, $p^2 + 2 p q + q^2 = 1$, and $p + q = 1$ where $p$ is the frequency of the\
\ dominant allele, $q$ is the frequency of the recessive allele, and $p^2$, $q^2$,\
\ and $2pq$ are the frequencies of dominant homozygous, recessive homozygous, and\
\ heterozygous individuals, respectively. ​The frequency of the recessive allele\
\ (q) is $\\sqrt{\frac{1}{400}} = 0.05$. We have $p = 1 - q = 0.95$. The frequency\
\ of heterozygous individuals is $2pq = 2 \\cdot 0.05 \\cdot 0.95 = 0.095$. The\
\ number of heterozygous individuals is equal to the frequency of heterozygous individuals\
\ times the size of the population, or $0.095 * 400 = 38$. So we end up with 38/400.\
\ The answer is (D).\n\nQ: According to the pressure-flow model of movement of phloem\
\ contents, photosynthate movement from source to sink is driven by\n(A) an ATP-dependent\
\ pressure-flow pump (B) a water-pressure potential gradient (C) transpiration (D)\
\ apoplastic diffusion\nA: Let's think step by step. It is a gradient in water pressure\
\ that induces the movement of phloem content, which refers to answer (B). The mechanism\
\ of movement does not rely on metabolism, which rules out (A). Transpiration refers\
\ to the exhalation of water vapor through plant stomata, and is also not related,\
\ which rules out (C). While the apoplastic pathway is one of two main pathways\
\ for water transport in plants, it is not central to the pressure flow model, which\
\ rules out (D). The answer is (B).\n\nQ: Which of the following contain DNA sequences\
\ required for the segregation of chromosomes in mitosis and meiosis?\n(A) Telomeres\
\ (B) Centromeres (C) Nucleosomes (D) Spliceosomes\nA: Let's think step by step.\
\ The genetic material in Telomeres is not used, which rules out (A). Nucleosomes\
\ are the repeating subunit that comprises chromatin packed in a cell nucleus, and\
\ do not specifically refer to DNA sequences necessary for segregating chromosomes\
\ in cell division, which rules out (C). A spliceosome is a large ribonucleoprotein\
\ that removes introns from transcribed pre-mRNA rather than governing chromosome\
\ segregation. Centromeres are directly responsible for segregating chromosomes\
\ in cell division. The answer is (B).\n\nQ: The presence of homologous structures\
\ in two different organisms, such as the humerus in the front limb of a human and\
\ a bird, indicates that\n(A) the human and bird are polyphyletic species (B) a\
\ human's and bird's evolution is convergent (C) the human and bird belong to a\
\ clade (D) the human and bird developed by analogy\nA: Let's think step by step.\
\ Polyphyletic species are organisms that are grouped due to having similar characteristics\
\ but which do not have a common ancestor. This is not the case for humans and birds,\
\ which rules out (A). Convergent evolution refers to the indepdendent development\
\ of similar features in different species at different periods, which is also not\
\ the case for humans and birds, which rules out (B). Analogy refers to the superficial\
\ resemblance of structures that have different origins, which is not the case for\
\ the human and bird forearms, which rules out (D). Humans and birds do belong to\
\ the same clade - a group of organisms composed of a common ancestor. The answer\
\ is (C)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_biology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_chemistry"
"description": "The following are multiple choice questions (with answers) about college\
\ chemistry.\n\nQ: 3 Cl−(aq) + 4 CrO_4^2−(aq) + 23 H+(aq) → 3 HClO2(aq) + 4 Cr3+(aq)\
\ + 10 H2O(l). In the reaction shown above, Cl−(aq) behaves as\n(A) an acid (B)\
\ a base (C) a catalyst (D) a reducing agent\nA: Let's think step by step. A molecule\
\ that behaves as a base accepts an H+ ion (or proton) from another molecule, whereas\
\ a molecule that behaves as an acid donates an H+ ion (or proton) to another molecule.\
\ Neither of these is the case for Cl in this reaction, which rules out (A) and\
\ (B). A catalyst is a substance that only accelerates a reaction without itself\
\ undergoing chemical change, which is not the case here. This rules out (C). Instead,\
\ the $Cl^{-} molecules carry a negative charge, which they donate in the reaction\
\ to form 3 HClO2. This is the behavior of a reducing agent, or (D). The answer\
\ is (D).\n\nQ: Which of the following statements about the lanthanide elements\
\ is NOT true?\n(A) The most common oxidation state for the lanthanide elements\
\ is +3. (B) Lanthanide complexes often have high coordination numbers (> 6). (C)\
\ All of the lanthanide elements react with aqueous acid to liberate hydrogen. (D)\
\ The atomic radii of the lanthanide elements increase across the period from La\
\ to Lu.\nA: Let's think step by step. The atomic radii of the lanthanide elements\
\ in fact decrease across the period from La to Lu. Options (A), (B), and (C) are\
\ all true. This means that only (D) is NOT true. The answer is (D).\n\nQ: Which\
\ of the following lists the hydrides of group-14 elements in order of thermal stability,\
\ from lowest to highest?\n(A) PbH4 < SnH4 < GeH4 < SiH4 < CH4 (B) PbH4 < SnH4 <\
\ CH4 < GeH4 < SiH4 (C) CH4 < SiH4 < GeH4 < SnH4 < PbH4 (D) CH4 < PbH4 < GeH4 <\
\ SnH4 < SiH4\nA: Let's think step by step. The thermal stability of group-14 hydrides\
\ decreases as we move from the top of group 14 to the bottom. The order of elements\
\ in the group from top to bottom is C, Si, Ge, Sn, Pb. Therefore in order of increasing\
\ thermal stability we have PbH4, SnH4, GeH4, SiH4, and CH4, or answer (A). The\
\ answer is (A).\n\nQ: Predict the number of lines in the EPR spectrum of a solution\
\ of 13C-labelled methyl radical (13CH3•), assuming the lines do not overlap.\n\
(A) 4 (B) 3 (C) 6 (D) 24 (E) 8\nA: Let's think step by step. The electron paramagnetic\
\ resonance spectrum will be split by two forms of interactions. The first is the\
\ hyperfine interaction with the 13C (nuclear spin $I = \nrac{1}{2}$) which will\
\ split the spectrum into 2 lines. This will be further split into 4 lines by the\
\ interaction with three equivalent 1H nuclei. The total number of lines is therefore\
\ $2 \\cdot 4 = 8$. The answer is (E)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_chemistry"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_computer_science"
"description": "The following are multiple choice questions (with answers) about college\
\ computer science.\n\nQ: Which of the following regular expressions is equivalent\
\ to (describes the same set of strings as) (a* + b)*(c + d)?\n(A) a*(c + d)+ b(c\
\ + d)\n(B) a*(c + d)* + b(c + d)*\n(C) a*(c + d)+ b*(c + d)\n(D) (a + b)*c +(a\
\ + b)*d\nA: Let's think step by step. We know that:\n1. (X* + Y)* = (X + Y)*\n\
2. X(Y + Z)? = XY + XZ\nUsing equation 1 we can rewrite (a* + b)*(c + d)? as:\n\
3. (a + b)*(c + d)?\nUsing equation 2 we can rewrite equation 3 as:\n(a + b)*c +\
\ (a + b)*d The answer is (D).\n\nQ: The Singleton design pattern is used to guarantee\
\ that only a single instance of a class may be instantiated. Which of the following\
\ is (are) true of this design pattern?\nI. The Singleton class has a static factory\
\ method to provide its instance.\nII. The Singleton class can be a subclass of\
\ another class.\nIII. The Singleton class has a private constructor.\n(A) I only\n\
(B) II only\n(C) III only\n(D) I, II, and III\nA: Let's think step by step. Statement\
\ I is a correct statement about a Singleton, because a Singleton restricts instantiation\
\ to a single, static method. Statement II is also correct, because there is no\
\ inherent restriction regarding the inheritance of a Singleton. Statement III is\
\ also correct, because a Singletons must be instantiated only once, so its constructor\
\ is made private to prevent any construction except via its static factory method.\n\
Given these facts, statements I, II, and III are all correct. The answer is (D).\n\
\nQ: A certain pipelined RISC machine has 8 general-purpose registers R0, R1, .\
\ . . , R7 and supports the following operations:\nADD Rs1, Rs2, Rd (Add Rs1 to\
\ Rs2 and put the sum in Rd)\nMUL Rs1, Rs2, Rd (Multiply Rs1 by Rs2 and put the\
\ product in Rd)\nAn operation normally takes one cycle; however, an operation takes\
\ two cycles if it produces a result required by the immediately following operation\
\ in an operation sequence.\nConsider the expression AB + ABC + BC, where variables\
\ A, B, C are located in registers R0, R1, R2. If the contents of these three registers\
\ must not be modified, what is the minimum number of clock cycles required for\
\ an operation sequence that computes the value of AB + ABC + BC?\n(A) 5 (B) 6 (C)\
\ 7 (D) 8\nA: Let's think step by step. First, we are given that A is in R0, B is\
\ in R1, and C is in R2.\nNext, we can see that we must compute three multiplies\
\ (AB, BC, and ABC) and two adds (AB + ABC, (AB + ABC) + BC) to compute our final\
\ answer, resulting in a minimum of five clock cycles.\nNext, we can see that there\
\ is no way to avoid at least one pipeline stall when computing our final answer,\
\ because to compute our final sum we must wait at least one cycle for the results\
\ from the previous stage to be ready. Thus, our minimum number of cycles must be\
\ 6.\nWe can verify that we can create a solution that requires only six cycles\
\ as follows:\ncompute AB: MUL R0, R1, R3\ncompute BC: MUL R1, R2, R4\ncompute ABC:\
\ MUL R3, R4, R5\ncompute AB + BC: ADD R3, R4, R6\nSTALL\ncompute AB + ABC + BC:\
\ ADD R5, R6, R7\nSo there are 6 cycles. The answer is (B).\n\nQ: A compiler generates\
\ code for the following assignment statement.\nG := (A + B) * C - (D + E) * F\n\
The target machine has a single accumulator and a single-address instruction set\
\ consisting of instructions load, store, add, subtract, and multiply. For the arithmetic\
\ operations, the left operand is taken from the accumulator and the result appears\
\ in the accumulator. The smallest possible number of instructions in the resulting\
\ code is\n(A) 5 (B) 6 (C) 7 (D) 9\nA: Let's think step by step. We can compute\
\ the final answer with the following sequence of operations:\n1. LOAD D (accumulator\
\ = D)\n2. ADD E (accumulator = D+E)\n3. MUL F (accumulator = (D+E)*F)\n4. STORE\
\ X (X = (D+E)*F)\n5. LOAD A (accumulator = A)\n6. ADD B (accumulator = A+B)\n\
7. MUL C (accumulator = (A+B)*C)\n8. SUB X (accumulator = (A+B)*C - (D+E)*F)\n\
9. STORE G (G = (A+B)*C - (D+E)*F)\nThis sequence takes 9 instructions. The answer\
\ is (D).\n\nQ: Consider a computer design in which multiple processors, each with\
\ a private cache memory, share global memory using a single bus. This bus is the\
\ critical system resource. Each processor can execute one instruction every 500\
\ nanoseconds as long as memory references are satisfied by its local cache. When\
\ a cache miss occurs, the processor is delayed for an additional 2,000 nanoseconds.\
\ During half of this additional delay, the bus is dedicated to serving the cache\
\ miss. During the other half, the processor cannot continue, but the bus is free\
\ to service requests from other processors. On average, each instruction requires\
\ 2 memory references. On average, cache misses occur on 1 percent of references.\
\ What proportion of the capacity of the bus would a single processor consume, ignoring\
\ delays due to competition from other processors?\n(A) 1/50 (B) 1/27 (C) 1/25 (D)\
\ 2/27\nA: Let's think step by step. We know that each instruction requires two\
\ memory references per instruction, and that there is an average cache miss rate\
\ of one percent.\nThus a given processor has:\n(1 cache miss / 100 references)\
\ * (2 references / instruction) =\n(2 cache misses / 100 instructions), so:\nmisses_per_instruction\
\ = 1 cache miss / 50 instructions.\nNext, we know that each instruction requires\
\ 500 nanoseconds when there is no cache miss, and 500 + 2000 = 2500 nanoseconds\
\ when there is a cache miss. Thus:\n50 instructions / (49 * 500) + (1 * 2500) nanoseconds,\
\ so:\ninstructions_per_ns = 50 instructions / 27000 nanoseconds.\nNow, we know\
\ that each cache miss locks the bus for half of the 2000 nanosecond cache miss\
\ delay, or 1000 nanoseconds, so:\nlock_ns_per_miss = 1000 nanoseconds / cache miss.\n\
Thus we can see that on average a single processor will lock the bus for:\nlock_ns_per_miss\
\ * misses_per_instruction * instructions_per_ns =\n(1000 nanoseconds / cache miss)\
\ * (1 cache miss / 50 instructions) * (50 instructions / 27000 nanoseconds) = 1000\
\ * (1/50) * (50/27000) = 1000/27000 = 1/27. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_computer_science"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_mathematics"
"description": "The following are multiple choice questions (with answers) about college\
\ mathematics.\n\nQ: Let V be the set of all real polynomials p(x). Let transformations\
\ T, S be defined on V by T:p(x) -> xp(x) and S:p(x) -> p'(x) = d/dx p(x), and interpret\
\ (ST)(p(x)) as S(T(p(x))). Which of the following is true?\n(A) ST = 0 (B) ST =\
\ T (C) ST = TS (D) ST - TS is the identity map of V onto itself.\nA: Let's think\
\ step by step. For a given polynomial $p$ we have\n\\[ST(p) = (xp(x))’ = p(x) +\
\ xp’(x)\\]\nand\n\\[TS(p) = xp’(x).\\]\nHence \\[ST(p) - TS(p) = p(x) + xp’(x)\
\ - xp’(x).\\] The answer is (D).\n\nQ: Suppose that f(1 + x) = f(x) for all real\
\ x. If f is a polynomial and f(5) = 11, then f(15/2)\n(A) -11 (B) 0 (C) 11 (D)\
\ 33/2\nA: Let's think step by step. The only polynomial so that $f(1 + x) = f(x)$\
\ is a constant polynomial. Hence $f(5) = 11 = f(15/2)$. The answer is (C).\n\n\
Q: Let A be a real 2x2 matrix. Which of the following statements must be true?\n\
I. All of the entries of A^2 are nonnegative.\nII. The determinant of A^2 is nonnegative.\n\
III. If A has two distinct eigenvalues, then A^2 has two distinct eigenvalues.\n\
(A) I only (B) II only (C) III only (D) II and III only\nA: Let's think step by\
\ step. We have \\[ det(A^2) = (det(A))^2 \\geq 0,\\] hence II holds.\nIII is false:\
\ as a counterexample take a diagonal matrix with -1 and 1 on the diagonal. Then\
\ $A^2$ is the identity matrix. The answer is (B).\n\nQ: Let A be the set of all\
\ ordered pairs of integers (m, n) such that 7m + 12n = 22. What is the greatest\
\ negative number in the set B = {m + n : (m, n) \\in A}?\n(A) -5 (B) -4 (C) -3\
\ (D) -2\nA: Let's think step by step. We have 12n = 22 - 7m and one of the solutions\
\ is $m = -2$, $n = 3$. Then $m + n = 1$, hence we need to look for smaller $m$\
\ in order to make $m + n$ negative. The next solution is $m = -14$ and $n = 10$.\
\ For smaller $m$ we have $m + n$ smaller than $-4$. The answer is (B).\n\nQ: A\
\ tank initially contains a salt solution of 3 grams of salt dissolved in 100 liters\
\ of water. A salt solution containing 0.02 grams of salt per liter of water is\
\ sprayed into the tank at a rate of 4 liters per minute. The sprayed solution is\
\ continually mixed with the salt solution in the tank, and the mixture flows out\
\ of the tank at a rate of 4 liters per minute. If the mixing is instantaneous,\
\ how many grams of salt are in the tank after 100 minutes have elapsed?\n(A) 2\
\ (B) 2 - e^-2 (C) 2 + e^-2 (D) 2 + e^-4\nA: Let's think step by step. For all $t\
\ \\in \\mathbb{R}$, let $s(t)$ denote the number grams of salt in the tank at the\
\ $t$ minute mark. Then $s(0) = 3$.\nWe use $s$ and $s(t)$ interchangeably. We also\
\ use $s^{\\prime}$ and $s^{\\prime}(t)$ interchangeably. The solution sprayed into\
\ the tank adds $(0.02) 4=2 / 25$ grams of salt per minute. There are always 100\
\ liters of liquid in the tank, containing $s$ grams of salt. So the density of\
\ salt in the tank is $s / 100$ grams per liter. The flow of water out of the tank\
\ therefore subtracts $4(s / 100)=s / 25$ grams of salt per minute. Then, for all\
\ $t \\in \\mathbb{R}$, we have $s^{\\prime}(t)=(2 / 25)-(s / 25)=(2-s) / 25$, and\
\ so $[s(t)=2] \\Rightarrow\\left[s^{\\prime}(t)=0\right]$. For all $t \\in \\mathbb{R}$,\n\
$$\n\frac{d}{d t}[\\ln (s-2)]=\frac{s^{\\prime}}{s-2}=\frac{-1}{25}=\frac{d}{d t}\\\
left[-\frac{t}{25}\right] .\n$$\nChoose $C \\in \\mathbb{R}$ such that, for all\
\ $t \\in \\mathbb{R}, \\ln ((s(t)-2))=-[t / 25]+C$. Let $K:=e^{C}$. Then, for all\
\ $t \\in \\mathbb{R}$, we have $(s(t))-2=K e^{-t / 25}$, and so $s(t)=2+K e^{-t\
\ / 25}$. Then $3=s(0)=2+K e^{0}=2+K$, so $K=1$. Then $s(100)=2+K e^{-100 / 25}=2+1\
\ \\cdot e^{-4}=2+e^{-4}$. The answer is (D)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_medicine"
"description": "The following are multiple choice questions (with answers) about college\
\ medicine.\n\nQ: An expected side effect of creatine supplementation is:\n(A) muscle\
\ weakness. (B) gain in body mass. (C) muscle cramps. (D) loss of electrolytes.\n\
A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
\ Creatine supplementation is a dietary supplement that results in body mass gain.\
\ The answer is (B).\n\nQ: Which of the following is not a true statement?\n(A)\
\ Muscle glycogen is broken down enzymatically to glucose-1-phosphate (B) Elite\
\ endurance runners have a high proportion of Type I fibres in their leg muscles\
\ (C) Liver glycogen is important in the maintenance of the blood glucose concentration\
\ (D) Insulin promotes glucose uptake by all tissues in the body\nA: Let's think\
\ step by step. We refer to Wikipedia articles on medicine for help. Let’s solve\
\ this step by step and go over each choice: \n(A) “Muscle glycogen is broken down\
\ enzymatically to glucose-1-phosphate”: This is a correct statement.\n(B) “Elite\
\ endurance runners have a high proportion of Type I fibres in their leg muscles”:\
\ This is a correct statement.\n(C) “Liver glycogen is important in the maintenance\
\ of the blood glucose concentration”: This is a correct statement. \n(D) “Insulin\
\ promotes glucose uptake by all tissues in the body”: This is not a correct statement,\
\ because insulin promotes glucose uptake by the liver, adipose tissue, and muscle,\
\ but not all tissues. For instance, the tissues in the brain and red blood cells\
\ are not affected by insulin. The answer is (D).\n\nQ: A high school science teacher\
\ fills a 1 liter bottle with pure nitrogen and seals the lid. The pressure is 1.70\
\ atm, and the room temperature is 25°C. Which two variables will both increase\
\ the pressure of the system, if all other variables are held constant?\n(A) Increasing\
\ temperature, increasing moles of gas (B) Increasing temperature, increasing volume\
\ (C) Decreasing volume, decreasing temperature (D) Decreasing moles of gas, increasing\
\ volume\nA: Let's think step by step. We refer to Wikipedia articles on medicine\
\ for help. The relevant equation for this is the ideal gas law: PV=nRT. To increase\
\ the pressure of the system (P), then either n (number of moles of the gas) or\
\ T (temperature) have to increase. The answer is (A).\n\nQ: In a genetic test of\
\ a newborn, a rare genetic disorder is found that has X-linked recessive transmission.\
\ Which of the following statements is likely true regarding the pedigree of this\
\ disorder?\n(A) All descendants on the maternal side will have the disorder. (B)\
\ Females will be approximately twice as affected as males in this family. (C) All\
\ daughters of an affected male will be affected. (D) There will be equal distribution\
\ of males and females affected.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on medicine for help. Let’s solve this step by step. Let's recall first\
\ that females have two X chromosomes, while males have one X and one Y chromosome.\
\ This is an important fact we need to know before answering this question. \nBecause\
\ a male can only pass his only one X chromosome to a daughter, if he is affected\
\ by this rare genetic disorder, then we know for sure that he will pass this rare\
\ genetic disorder to all his future-born daughters. Therefore, “(C): All daughters\
\ of an affected male will be affected” is a correct statement. The answer is (C).\n\
\nQ: Glucose is transported into the muscle cell:\n(A) via protein transporters\
\ called GLUT4. (B) only in the presence of insulin. (C) via hexokinase. (D) via\
\ monocarbylic acid transporters.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on medicine for help. Glucose (also known as the blood sugar) is the\
\ main sugar found in the human body. It is transported into the muscle cell via\
\ diffusion through protein transporters called GLUT4. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_medicine"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "college_physics"
"description": "The following are multiple choice questions (with answers) about college\
\ physics.\n\nQ: A refracting telescope consists of two converging lenses separated\
\ by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification\
\ of the telescope is\n(A) 4 (B) 5 (C) 6 (D) 20\nA: Let's think step by step. In\
\ a refracting telescope, if both lenses are converging, the focus of both lenses\
\ must be between the two lenses, and thus the focal lengths of the two lenses must\
\ add up to their separation. Since the focal length of one lens is 20 cm, the focal\
\ length of the other must be 80 cm. The magnification is the ratio of these two\
\ focal lengths, or 4. The answer is (A).\n\nQ: The muon decays with a characteristic\
\ lifetime of about 10^-6 second into an electron, a muon neutrino, and an electron\
\ antineutrino. The muon is forbidden from decaying into an electron and just a\
\ single neutrino by the law of conservation of\n(A) charge (B) mass (C) energy\
\ and momentum (D) lepton number\nA: Let's think step by step. Lepton number must\
\ be conserved, meaning the total number of leptons minus the number of antileptons.\
\ If a muon decays into an electron and a single neutrino, the total lepton number\
\ would go from one to two, violating lepton number conservation. The answer is\
\ (D).\n\nQ: One end of a Nichrome wire of length 2L and cross-sectional area A\
\ is attached to an end of another Nichrome wire of length L and cross- sectional\
\ area 2A. If the free end of the longer wire is at an electric potential of 8.0\
\ volts, and the free end of the shorter wire is at an electric potential of 1.0\
\ volt, the potential at the junction of the two wires is most nearly equal to\n\
(A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V\nA: Let's think step by step. This is a\
\ simple voltage divider problem, where the longer wire has a resistance four times\
\ that of the shorter end. So the voltage divider ratio is 1 / 5, meaning that the\
\ potential in the middle is 1.0 V + (8.0 V - 1.0 V) * 1/5 = 2.4 V. The answer is\
\ (A).\n\nQ: A refracting telescope consists of two converging lenses separated\
\ by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification\
\ of the telescope is\n(A) 4 (B) 5 (C) 6 (D) 20\nA: Let's think step by step. In\
\ a refracting telescope, if both lenses are converging, the focus of both lenses\
\ must be between the two lenses, and thus the focal lengths of the two lenses must\
\ add up to their separation. Since the focal length of one lens is 20 cm, the focal\
\ length of the other must be 80 cm. The magnification is the ratio of these two\
\ focal lengths, or 4. The answer is (A).\n\nQ: For which of the following thermodynamic\
\ processes is the increase in the internal energy of an ideal gas equal to the\
\ heat added to the gas?\n(A) Constant temperature (B) Constant volume (C) Constant\
\ pressure (D) Adiabatic\nA: Let's think step by step. Heat added to the gas can\
\ go into the gases internal energy or work done against an external force. However,\
\ if the volume of the gas container is constant, no work will be done (since work\
\ is pressure times change in volume). So, at constant volume, all of the heat goes\
\ into the internal energy. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_physics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "computer_security"
"description": "The following are multiple choice questions (with answers) about computer\
\ security.\n\nQ: SHA-1 has a message digest of\n(A) 160 bits (B) 512 bits (C) 628\
\ bits (D) 820 bits\nA: Let's think step by step. Since SHA-1 is a hash function\
\ which takes an input and produces a 160-bit (20-byte) hash value, its message\
\ digest is 160 bits. The answer is (A).\n\nQ: _____________ can modify data on\
\ your system – so that your system doesn’t run correctly or you can no longer access\
\ specific data, or it may even ask for ransom in order to give your access.\n(A)\
\ IM – Trojans (B) Backdoor Trojans (C) Trojan-Downloader (D) Ransom Trojan\nA:\
\ Let's think step by step. The system is asking for trojans, which are for ransom,\
\ which means ransom trojan. The answer is (D).\n\nQ: What is ethical hacking?\n\
(A) \"Hacking\" ethics so they justify unintended selfish behavior (B) Hacking systems\
\ (e.g., during penetration testing) to expose vulnerabilities so they can be fixed,\
\ rather than exploited (C) Hacking into systems run by those whose ethics you disagree\
\ with (D) A slang term for rapid software development, e.g., as part of hackathons\n\
A: Let's think step by step. Ethical hacking is a process of detecting vulnerabilities\
\ in an application, system, or organization's infrastructure that an attacker can\
\ use to exploit an individual or organization. They use this process to prevent\
\ cyberattacks and security breaches by lawfully hacking into the systems and looking\
\ for weak points. The answer is (B).\n\nQ: The ____________ is anything which your\
\ search engine cannot search.\n(A) Haunted web (B) World Wide Web (C) Surface web\
\ (D) Deep Web\nA: Let's think step by step. The search engine searches on the Surface\
\ Web, which is the portion of the world wide web which is visible so (B,C) are\
\ wrong. The Haunted Web doesn’t correspond to an internet concept. The Deep Web\
\ is the part of the World Wide Web which is not indexed. The answer is (D).\n\n\
Q: Exploitation of the Heartbleed bug permits\n(A) overwriting cryptographic keys\
\ in memory (B) a kind of code injection (C) a read outside bounds of a buffer (D)\
\ a format string attack\nA: Let's think step by step. The Heartbleed Bug is a serious\
\ vulnerability in the popular OpenSSL cryptographic software library. Heartbleed\
\ resulted from improper input validation (due to a missing bounds check) in the\
\ implementation of the TLS heartbeat extension. The vulnerability was classified\
\ as a buffer over-read, a situation where more data can be read than should be\
\ allowed. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_computer_security"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "conceptual_physics"
"description": "\nThe following are multiple choice questions (with answers) about\
\ conceptual physics.\n\nQ: Colors in a soap bubble result from light\n(A) converted\
\ to a different frequency (B) deflection (C) interference (D) polarization\nA:\
\ Let's think step by step. In a soap bubble film, the light bounces between the\
\ two soap-air interfaces many times, interfering with itself constructively or\
\ destructively depending on the width of the film. This results in different colors\
\ being visible. The answer is (C).\n\nQ: Compared with the mass of a uranium atom\
\ undergoing fission, the combined masses of the products after fission are\n(A)\
\ less (B) more (C) the same (D) zero\nA: Let's think step by step. Fission releases\
\ energy, which comes from the rest mass of its initial nucleus. Thus the mass of\
\ the products is less than the mass of the reactant uranium nucleus. The answer\
\ is (A).\n\nQ: Things that are equivalent according to the equivalence principle\
\ are\n(A) space and time. (B) a traveling twin and a stay-at-home twin. (C) gravity\
\ and acceleration. (D) mass and energy.\nA: Let's think step by step. Einstein’s\
\ famous equivalence principle states that gravity and acceleration are equivalent.\
\ The answer is (C).\n\nQ: Which of these three elements has the most mass per nucleon?\n\
(A) Hydrogen (B) Iron (C) Uranium (D) Same in each\nA: Let's think step by step.\
\ Due to nuclear binding energy, the mass of an atomic nucleus is less than the\
\ sum of individual masses of the free constituent protons and neutrons; this is\
\ known as the mass defect. Hydrogen has no mass defect because it has only a single\
\ nucleon, so it will have the most mass per nucleon. The answer is (A).\n\nQ: A\
\ model airplane flies slower when flying into the wind and faster with wind at\
\ its back. When launched at right angles to the wind a cross wind its groundspeed\
\ compared with flying in still air is\n(A) the same (B) greater (C) less (D) either\
\ greater or less depending on wind speed\nA: Let's think step by step. The plane’s\
\ speed in the direction of the wind is greater than it would be in the absence\
\ of wind, and its direction orthogonal to the wind is the same as it would be in\
\ the absence of the wind. The total speed, which is these two components added\
\ in quadrature, is thus greater than the speed in still air. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_conceptual_physics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "econometrics"
"description": "The following are multiple choice questions (with answers) about econometrics.\n\
\nQ: Suppose now that a researcher wishes to use information criteria to determine\
\ the optimal lag length for a VAR. 500 observations are available for the bi-variate\
\ VAR, and the values of the determinant of the variance-covariance matrix of residuals\
\ are 0.0336, 0.0169, 0.0084, and 0.0062 for 1, 2, 3, and 4 lags respectively. What\
\ is the optimal model order according to Akaike's information criterion?\n(A) 1\
\ lag (B) 2 lags (C) 3 lags (D) 4 lags\nA: Let's think step by step. We refer to\
\ Wikipedia articles on econometrics for help. Let’s solve this problem step by\
\ step. First of all, let’s recall that for a given set of data, Akaike's information\
\ criterion (AIC) allows us to measure how well a statistical model fits the data;\
\ it is an estimator of prediction error. Here in this problem we will need to use\
\ the formula ln(det(sigma_hat)) + (2 * k / T) to determine the values of Akaike’s\
\ criterion, where ln denotes the natural log function, det the determinant function,\
\ k the total number of parameters in total (across both equations), and T the number\
\ of observations (which, in this case, is equal to 500). For 1 lag, the number\
\ of parameters in total is equal to 6; for 2 lags, it is 10; for 3 lags, it is\
\ 14; and for 4 lags, it is 18. Now, let’s calculate the values of the criterion\
\ for each lag:\n(A) 1 lag: ln(0.0336) + (2 * 6 / 500) = ln(0.0336) + (12 / 500)\
\ = -3.369\n(B) 2 lags: ln(0.0169) + (2 * 10 / 500) = ln(0.0169) + (20 / 500) =\
\ -4.040\n(C) 3 lags: ln(0.0084) + (2 * 14 / 500) = ln(0.0084) + (28 / 500) =-4.724\n\
(D) 4 lags: ln(0.0062) + (2 * 18 / 500) = ln(0.0062) + (36 / 500) =-5.011\nBecause\
\ the optimal model order according to AIC minimizes the information criterion,\
\ the answer should be the one with the lowest value. In this case, (D) has the\
\ lowest value. The answer is (C).\n\nQ: Consider the following AR(1) model with\
\ the disturbances having zero mean and unit variance\nyt = 0.2 + 0.4 yt-1 + ut\n\
The (unconditional) mean of y will be given by\n(A) 0.2 (B) 0.4 (C) 0.5 (D) 0.33\n\
A: Let's think step by step. We refer to Wikipedia articles on econometrics for\
\ help. Let’s solve this problem step by step. If we have a an AR(1) model with\
\ the disturbances having zero mean and unit variance, then the unconditional mean\
\ of y is equal to the following:\nunconditional mean of y = (the intercept term)\
\ / (1 - autoregressive coefficient)\nWe know that the intercept term is 0.2 and\
\ the autoregressive coefficient is 0.4; thus, we have:\nunconditional mean of y\
\ = (0.2) / (1 - 0.4) = (0.2) / (0.6) = 2 / 6 = 1 / 3, which is approximately 0.33.\
\ That means that the answer should be (D) 0.33. The answer is (D).\n\nQ: What would\
\ be then consequences for the OLS estimator if heteroscedasticity is present in\
\ a regression model but ignored?\n(A) It will be biased (B) It will be inconsistent\
\ (C) It will be inefficient (D) All of (a), (b) and (c) will be true.\nA: Let's\
\ think step by step. We refer to Wikipedia articles on econometrics for help. Heteroscedasticity\
\ refers to the condition where the variance of the error terms is not constant\
\ across multiple observations. If heteroscedasticity is present in a regression\
\ model, then the coefficient estimates in the OLS estimator will be not only unbiased\
\ and consistent but also inefficient. Because (A) and (B) are incorrect choices\
\ and (C) is a correct choice, (D) cannot be the right answer. Ultimately, (C) is\
\ the only true choice. The answer is (C).\n\nQ: Suppose that a test statistic has\
\ associated with it a p-value of 0.08. Which one of the following statements is\
\ true?\n(i) If the size of the test were exactly 8%, we would be indifferent between\
\ rejecting and not rejecting the null hypothesis\n(ii) The null would be rejected\
\ if a 10% size of test were used\n(iii) The null would not be rejected if a 1%\
\ size of test were used\n(iv) The null would be rejected if a 5% size of test were\
\ used.\n(A) (ii) and (iv) only (B) (i) and (iii) only (C) (i), (ii), and (iii)\
\ only (D) (i), (ii), (iii), and (iv).\nA: Let's think step by step. We refer to\
\ Wikipedia articles on econometrics for help. Let’s reason about each of the options.\n\
(i) is a true statement.\n(ii) is a true statement.\n(iii) is a true statement.\n\
(iv) is not a true statement. Thus, (i), (ii), and (iii) are true. The answer is\
\ (C).\n\nQ: For a stationary autoregressive process, shocks will\n(A) Eventually\
\ die away (B) Persist indefinitely (C) Grow exponentially (D) Never occur\nA: Let's\
\ think step by step. We refer to Wikipedia articles on econometrics for help. This\
\ is a formal logic problem about stationally process. For a stationary autoregressive\
\ process, shocks will eventually die away. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_econometrics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "electrical_engineering"
"description": "\nThe following are multiple choice questions (with answers) about\
\ electrical engineering.\n\nQ: A point pole has a strength of 4π * 10^-4 weber.\
\ The force in newtons on a point pole of 4π * 1.5 * 10^-4 weber placed at a distance\
\ of 10 cm from it will be\n(A) 15 N. (B) 20 N. (C) 7.5 N. (D) 3.75 N.\nA: Let's\
\ think step by step. The force between two point poles is given by m_1m_2/(mu_0\
\ 4 \\pi r^2), in analogy to Coulomb’s law. Plugging in the values given in the\
\ question, we calculate that the force is approximately 15 N. The answer is (A).\n\
\nQ: The coil of a moving coil meter has 100 turns, is 40 mm long and 30 mm wide.\
\ The control torque is 240*10-6 N-m on full scale. If magnetic flux density is\
\ 1Wb/m2 range of meter is\n(A) 1 mA. (B) 2 mA. (C) 3 mA. (D) 4 mA.\nA: Let's think\
\ step by step. The torque on a coil in a uniform magnetic field is given by BANI,\
\ where B is the magnetic flux density, A is the area of the coil, N is the number\
\ of turns, and I is the current. So we have that I = (Torque)/(BAN), or 240e-6/(1200e-6\
\ * 100 * 1) = 2e-3. The answer is (B).\n\nQ: In an SR latch built from NOR gates,\
\ which condition is not allowed\n(A) S=0, R=0 (B) S=0, R=1 (C) S=1, R=0 (D) S=1,\
\ R=1\nA: Let's think step by step. An SR latch is a set-reset latch; in the case\
\ where S=1 and R=1, the circuit has no stable state; instead a race condition will\
\ be produced within the circuit, so the device will be in an undefined state. So\
\ S=1, R=1 is an illegal input. The answer is (D).\n\nQ: Two long parallel conductors\
\ carry 100 A. If the conductors are separated by 20 mm, the force per meter of\
\ length of each conductor will be\n(A) 100 N. (B) 0.1 N. (C) 1 N. (D) 0.01 N.\n\
A: Let's think step by step. The magnetic force-per-length between two current-carrying\
\ conductors is given by \\mu_0 I_1 I_2 / (2 \\pi r), where $r$ is the separation\
\ distance and I_1 and I_2 are the currents. Plugging in 100 A for I_1 and I_2,\
\ and 20 mm for r, gives 0.1 N. The answer is (B).\n\nQ: In a 2 pole lap winding\
\ dc machine , the resistance of one conductor is 2Ω and total number of conductors\
\ is 100. Find the total resistance\n(A) 200Ω (B) 100Ω (C) 50Ω (D) 10Ω\nA: Let's\
\ think step by step. In lap winding, effectively two resistors are connected in\
\ parallel, so the actual resistance of each pair is 1 Ohm. Since we have 50 pairs,\
\ we get a total resistance of 50 Ohms. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_electrical_engineering"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "elementary_mathematics"
"description": "The following are multiple choice questions (with answers) about elementary\
\ mathematics.\n\nQ: Olivia used the rule \"Add 11\" to create the number pattern\
\ shown below. 10, 21, 32, 43, 54. Which statement about the number pattern is true?\n\
(A) The 10th number in the pattern will be an even number.\n(B) The number pattern\
\ will never have two even numbers next to each other.\n(C) The next two numbers\
\ in the pattern will be an even number then an odd number.\n(D) If the number pattern\
\ started with an odd number then the pattern would have only odd numbers in it.\n\
A: Let's think step by step. Choice A is incorrect because every even-numbered term\
\ in the pattern is odd, and 10 is an even number. Choice B is correct, because\
\ adding an odd number (in this case 11) to an odd number produces an even number,\
\ and adding an odd number to an even number produces an odd number. Thus the terms\
\ in the pattern will alternate between odd and even, so there will never be two\
\ even numbers next to each other. Choice C is incorrect because the last term in\
\ the example is even (54), and we know that the terms will alternate between even\
\ and odd. Choice D is incorrect because the terms in the pattern will alternate\
\ between odd and even, regardless of the value of the first term. The answer is\
\ (B).\n\nQ: The population of the city where Michelle was born is 145,826. What\
\ is the value of the 5 in the number 145,826?\n(A) 5 thousands\n(B) 5 hundreds\n\
(C) 5 tens\n(D) 5 ones\nA: Let's think step by step. Choice A is correct, because\
\ there are three digits following the 5, so\nthe 5 is in the thousands place. Thus\
\ the other choices are incorrect. The answer is (A).\n\nQ: A store sells 107 different\
\ colors of paint. They have 25 cans of each color in storage. The number of cans\
\ of paint the store has in storage can be found using the expression below. 107\
\ × 25. How many cans of paint does the store have in storage?\n(A) 749\n(B) 2,675\n\
(C) 2,945\n(D) 4,250\nA: Let's think step by step. We can calculate 107 x 25 = (100\
\ x 25) + (7 x 25) = 2500 + 175 = 2675. The answer is (B).\n\nQ: A total of 30 players\
\ will play basketball at a park. There will be exactly 5 players on each team.\
\ Which statement correctly explains how to find the number of teams needed?\n(A)\
\ Add 5 to 30 to find 35 teams.\n(B) Divide 30 by 5 to find 6 teams.\n(C) Multiply\
\ 30 and 5 to find 150 teams.\n(D) Subtract 5 from 30 to find 25 teams.\nA: Let's\
\ think step by step. We want to find the number of teams. We know that there are\
\ 5 players/team, and 30 players. Thus to get the number of teams we divide players\
\ by players/team, so 30 players / 5 players/team = 6 teams. The answer is (B).\n\
\nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n(B) (5 x 5)\
\ + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think step by\
\ step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5 x 4). The\
\ answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_elementary_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "formal_logic"
"description": "The following are multiple choice questions (with answers) about formal\
\ logic.\n\nQ: Which of the given formulas of PL is the best symbolization of the\
\ following sentence?\nTurtles live long lives and are happy creatures, unless they\
\ are injured.\n(A) (L • H) ≡ I (B) (L • H) ∨ I (C) L • (H ∨ I) (D) L • (H ⊃ R).\n\
A: Let's think step by step. We refer to Wikipedia articles on formal logic for\
\ help. Let’s solve this step by step. Let “L” denote “living long”, H “being happy”,\
\ and “I” “being injured”. Now, consider each choice:\n(A) means (living long AND\
\ being happy) is equivalent to (being injured). \n(B) means (living long AND being\
\ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\
\ injured). \n(D) means (living long) AND (being happy implies being R), but what\
\ R denotes is not clear.\nObviously, (B) is the best symbolization of the original\
\ sentence. The answer is (B).\n\nQ: Select the best translation into predicate\
\ logic.George borrows Hector's lawnmower. (g: George; h: Hector; l: Hector's lawnmower;\
\ Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh (D) Bghl\nA: Let's think\
\ step by step. We refer to Wikipedia articles on formal logic for help. Let’s solve\
\ this step by step. We are told that “Bxyx” means “x borrows y from z”. We can\
\ rewrite “George borrows Hector's lawnmower” as “George borrows a lawnmower from\
\ Hector”, which can then be translated into predicate logic as “Bglh”. The answer\
\ “Bglh” appears in (C); therefore, (C) must be the correct answer. The answer is\
\ (C).\n\nQ: \nSelect the best English interpretation of the given arguments in\
\ predicate logic.\nDm\n(∀x)(Wx ⊃ ~Dx). \n(∀x)Wx ∨ Ag\t/ (∃x)Ax\n(A) Marina is a\
\ dancer. Some weaklings are not dancers. Either everything is a weakling or Georgia\
\ plays volleyball. So something plays volleyball. (B) Marina is a dancer. No weakling\
\ is a dancer. Everything is either a weakling or plays volleyball. So something\
\ plays volleyball. (C) Marina is a dancer. Some weaklings are not dancers. Everything\
\ is either a weakling or plays volleyball. So something plays volleyball. (D) Marina\
\ is a dancer. No weakling is a dancer. Either everything is a weakling or Georgia\
\ plays volleyball. So something plays volleyball.\nA: Let's think step by step.\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ by step. Let “D” denote “being a dancer”, “m” denote “Maria”, “g” denote “Georgia”,\
\ “W” denote “weakling”, “A” denote “playing volleyball”. Then, we have the following:\n\
1. Dm → Maria is a dance.\n2. (∀x)(Wx ⊃ ~Dx). → For all x, if x is a weakling, then\
\ x is not a dancer. In other words, no weakling is a dancer.\n3. (∀x)Wx ∨ Ag\t\
/ (∃x)Ax → For all x, x is a weakling or Georgia plays volleyball. So there exists\
\ an x that plays volleyball. \nOptions (A) and (C) do claim that some weaklings\
\ are not dancers, but the second argument strongly states that no weakling is a\
\ dancer. Thus, we can eliminate them. Option (B) omits the important detail about\
\ Georgia playing volleyball. Option (D) has all the details presented in the arguments\
\ and is the best English interpretation of the arguments. The answer is (D).\n\n\
Q: Select the best translation into predicate logic: No people drive on Mars.\n\
(A) ~Pd (B) (∀x)(Px ∨ ~Dx) (C) (∀x)(Px ⊃ ~Dx) (D) ~Dp\nA: Let's think step by step.\
\ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ by step. Let “P” denote “being on Mars” and “D” denote “driving on Mars”. Then\
\ let’s consider each option:\nOption (A): ~Pd → d is not on Mars.\nOption (B):\
\ (∀x)(Px ∨ ~Dx) → For all x, x is on Mars and x do not drive on Mars.\nOption (C):\
\ (∀x)(Px ⊃ ~Dx) → For all x, x is on Mars implies that x do not drive on Mars.\n\
Option (D): ~Dp: → p do not drive on Mars.\nOf all these options, Option (C) appears\
\ to be the best and most meaningful interpretation of the argument “No people drive\
\ on Mars.” The answer is (C)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_formal_logic"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "global_facts"
"description": "The following are multiple choice questions (with answers) about global\
\ facts.\n\nQ: As of 2017, how many of the world’s 1-year-old children today have\
\ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\nA: Let's\
\ think step by step. We refer to Wikipedia articles on global facts for help. According\
\ to data published by the World Health Organization, the nummber of 1-year-old\
\ children vaccinated in 2017 exceeds 80%. The answer is (A).\n\nQ: As of 2019,\
\ about what percentage of Americans agree that the state is run for the benefit\
\ of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think step by step.\
\ We refer to Wikipedia articles on global facts for help. In 2019, about 46% percentage\
\ of Americans agree that the state is run for the benefit of all the people. The\
\ answer is (B).\n\nQ: As of 2019, about what percentage of Russians say it is very\
\ important to have free media in our country without government/state censorship?\n\
(A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think step by step. We refer to Wikipedia\
\ articles on global facts for help. As of 2019, about 38% of Russians say it is\
\ very important to have free media in our country. The answer is (A).\n\nQ: As\
\ of 2015, since 1990 forests have ____ in Europe and have ____ in Africa and the\
\ Americas.\n(A) increased, increased (B) increased, decreased (C) decreased, increased\
\ (D) decreased, decreased\nA: Let's think step by step. We refer to Wikipedia articles\
\ on global facts for help. As of 2015, since 1990 forests have increased in Europe\
\ and have decreased in Africa and the Americas. The answer is (B).\n\nQ: Which\
\ of the following pairs of statements are both true (as of 2019)?\n(A) People tend\
\ to be optimistic about their own future and the future of their nation or the\
\ world. (B) People tend to be optimistic about their own future but pessimistic\
\ about the future of their nation or the world. (C) People tend to be pessimistic\
\ about their own future but optimistic about the future of their nation or the\
\ world. (D) People tend to be pessimistic about their own future and the future\
\ of their nation or the world.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on global facts for help. As of 2019, most people tend to be optimistic\
\ about their own future but pessimistic about the future of their nation or the\
\ world. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_global_facts"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "high_school_biology"
"description": "The following are multiple choice questions (with answers) about high\
\ school biology.\n\nQ: In animal cells, which of the following represents the most\
\ likely pathway that a secretory protein takes as it is synthesized in a cell?\n\
(A) Plasma membrane–Golgi apparatus–ribosome–secretory vesicle–rough ER (B) Ribosome–Golgi\
\ apparatus–rough ER–secretory vesicle–plasma membrane (C) Plasma membrane–Golgi\
\ apparatus–ribosome–secretory vesicle–rough ER (D) Ribosome–rough ER–Golgi apparatus–secretory\
\ vesicle–plasma membrane\nA: Let's think step by step. Protein synthesis starts\
\ at the ribosome, so we can eliminate (A) and (C). The ribosome is often in the\
\ endoplasmic reticulum and moves from there to the Golgi apparatus, where it is\
\ modified and packaged into a vesicle. The vesicle then floats to the plasma membrane\
\ and is secreted. The answer is (D).\n\nQ: A mutation in a bacterial enzyme changed\
\ a previously polar amino acid into a nonpolar amino acid. This amino acid was\
\ located at a site distant from the enzyme’s active site. How might this mutation\
\ alter the enzyme’s substrate specificity?\n(A) By changing the enzyme’s pH optimum\
\ (B) By changing the enzyme’s location in the cell (C) By changing the shape of\
\ the protein (D) An amino acid change away from the active site cannot alter the\
\ enzyme’s substrate specificity.\nA: Let's think step by step. A change in an amino\
\ acid leads to a change in the primary structure of the protein. A change in the\
\ primary structure may lead to a change in the secondary and the tertiary structure\
\ of the protein. A change in the tertiary structure means a change in the shape\
\ of the protein, so (C) has to be correct. Since the change does not affect the\
\ active site of the enzyme, we do not expect the activity of the enzyme to be affected.\
\ The answer is (C).\n\nQ: Which of the following is not a way to form recombinant\
\ DNA?\n(A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation\n\
A: Let's think step by step. The introduction of foreign DNA or RNA into bacteria\
\ or eukaryotic cells is a common technique in molecular biology and scientific\
\ research. There are multiple ways foreign DNA can be introduced into cells including\
\ transformation, transduction, conjugation, and transfection. In contrast, (A)\
\ is not a way to form DNA: during translation the ribosomes synthesize proteins\
\ from RNA. The answer is (A).\n\nQ: Homologous structures are often cited as evidence\
\ for the process of natural selection. All of the following are examples of homologous\
\ structures EXCEPT\n(A) the wings of a bird and the wings of a bat (B) the flippers\
\ of a whale and the arms of a man (C) the pectoral fins of a porpoise and the flippers\
\ of a seal (D) the forelegs of an insect and the forelimbs of a dog\nA: Let's think\
\ step by step. ​​Homologous structures are similar physical features in organisms\
\ that share a common ancestor ​​but different functions. Comparisons (B) and (C)\
\ are clearly homologous because they share a common ancestor and the structures\
\ serve different purposes. Bat wings and birg wings are also homologous, while\
\ they are both wings, the forelimbs serve different purposes. Insects and dogs\
\ are very far ancestors since one is vertebrate while the other is invertebrate\
\ and the forelimbs serve the same purpose, so they are not homologous. The answer\
\ is (D).\n\nQ: Which of the following is not known to be involved in the control\
\ of cell division?\n(A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast\
\ cells\nA: Let's think step by step. Normal cells move through the cell cycle in\
\ a regulated way. At the checkpoint stage, they use information about their own\
\ internal state and cues from the environment around them to decide whether to\
\ proceed with cell division. Cues like these act by changing the activity of core\
\ cell cycle regulators inside the cell. The most common regulators are cyclins\
\ and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division.\
\ The answer is (D)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_biology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "high_school_chemistry"
"description": "The following are multiple choice questions (with answers) about high\
\ school chemistry.\n\nQ: Which of the following is considered an acid anhydride?\n\
(A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3\nA: Let's think step by step. An acid anhydride\
\ is a compound that is derived by removing water from an acid. The chemical formula\
\ for water is H2O, which means that we need to determine which of these options,\
\ when combined with H2O, forms an acid. SO2, or Sulfur dioxide, when combined with\
\ H2O, makes H2SO4, or sulfuric acid. The answer is (C).\n\nQ: Which of the following\
\ is expected to be a polar molecule?\n(A) PCl4F (B) BF3 (C) CO2 (D) Si(CH3)4\n\
A: Let's think step by step. A polar molecule is one that has a slightly positive\
\ charge on one end of the molecule and a slightly negative charge on the other\
\ end. Boron trifluoride (BF3) has Boron as the center atom and three fluorine atoms\
\ attached to it; it is trigonal planar and symmetric, so it is nonpolar. Carbon\
\ Dioxide (CO2) has Carbon as the central atom with double bonds to two Oxygen atoms\
\ - this is also symmetrical and therefore nonpolar. The same is the case for tetramethyl\
\ silane (SI(CH3)4), which is a Silicon atom surrounded by four methyl groups. The\
\ structure of PCL4F is that Phosphorus is the central atom, attached to four chlorines\
\ and one fluorine atom. This is asymmetrical, and therefore has a net dipole and\
\ is expected to be a polar molecule. The answer is (A).\n\nQ: From the solubility\
\ rules, which of the following is true?\n(A) All chlorides, bromides, and iodides\
\ are soluble (B) All sulfates are soluble (C) All hydroxides are soluble (D) All\
\ ammonium-containing compounds are soluble\nA: Let's think step by step. The chlorides,\
\ bromides, and iodides of lead, silver, and mercury are not soluble in water. This\
\ rules out (A). The sulfates of lead, barium, and calcium are not soluble in water,\
\ which rules out (B). The hydroxides of any metal besides sodium, potassium, ammonium,\
\ calcium, and barium are insoluble. This rules out (C). Typically ammonium ions\
\ indicate a soluble ionic substance. The answer is (D).\n\nQ: A new compound is\
\ synthesized and found to be a monoprotic acid with a molar mass of 248 g/mol.\
\ When 0.0050 mol of this acid are dissolved in 0.500 L of water, the pH is measured\
\ as 3.89. What is the pKa of this acid?\n(A) 3.89 (B) 7.78 (C) 5.78 (D) 2.33\n\
A: Let's think step by step. Recall that $[A] = [H^{+}]$. Here, this is equal to\
\ $$10^{-3.89}$. Then we have $K_{a} = $\nrac{[H^{+}][A^{-}]}{[HA]} = \nrac{10^{-3.89}\
\ \\cdot 10^{-3.89}}{10^{-2}}. The resulting exponent is $-3.89 + (-3.89) - (-2)\
\ = 5.78$, therefore $K_a = 10^{-5.78}$. The $pK_a$ is the negative log of $K_a$,\
\ which is equal to $5.78$. The answer is (C).\n\nQ: A solution contains 2.00 mole\
\ of acetic acid, CH3COOH, and 1.00 mole of calcium acetate, Ca(CH3COO)2. The solution\
\ is able to resist the addition of a small amount of strong acid or strong base\
\ with only minor changes in the pH of the solution. Larger quantities of strong\
\ acid or strong base can cause a significant change in pH. How many moles of nitric\
\ acid, HNO3, may be added before the pH begins to change significantly?\n(A) 0.500\
\ mole (B) 1.00 mole (C) 2.00 mole (D) 3.00 mole\nA: Let's think step by step. We\
\ would like to compute the buffer capacity of this solution. First we write the\
\ equation for the ionization of the weak acid, in this case of acetic acid. $CH_{3}COOH\
\ (aq) + H_{2}O \nightarrow H_{3}O^{+} + CH3COO^{-}$. The conjugate base is therefore\
\ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\
\ base. Therefore the maximum amount of acid that can be added will be equal to\
\ the amount of acetate ion, or 2 moles. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_chemistry"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "high_school_computer_science"
"description": "The following are multiple choice questions (with answers) about high\
\ school computer science.\n\nQ: Which of the following is an example of the use\
\ of a device on the Internet of Things (IoT) ?\n(A) A car alerts a driver that\
\ it is about to hit an object. (B) A hiker uses a G P S watch to keep track of\
\ her position. (C) A refrigerator orders milk from an online delivery service when\
\ the milk in the refrigerator is almost gone. (D) A runner uses a watch with optical\
\ sensors to monitor his heart rate.\nA: Let's think step by step. The term Internet\
\ of Things (IoT) refers to common devices which are connected to the internet,\
\ enabling new functionality. Choice A is incorrect because it does not describe\
\ an internet connected device. In choice B, the watch is only described as having\
\ GPS functionality but no internet connectivity. Choice C describes a common device\
\ (a refrigerator) which has internet connectivity enabling new functionality (online\
\ ordering). Choice D does not mention internet connectivity for the watch, only\
\ optical sensors. The answer is (C).\n\nQ: Many Web browsers allow users to open\
\ anonymous windows. During a browsing session in an anonymous window, the browser\
\ does not record a browsing history or a list of downloaded files. When the anonymous\
\ window is exited, cookies created during the session are deleted. Which of the\
\ following statements about browsing sessions in an anonymous window is true?\n\
(A) The activities of a user browsing in an anonymous window will not be visible\
\ to people who monitor the user's network, such as the system administrator. (B)\
\ Items placed in a Web store's shopping cart for future purchase during the anonymous\
\ browsing session will not be saved on the user's computer. (C) A user will not\
\ be able to log in to e-mail or social media accounts during the anonymous browsing\
\ session. (D) A user browsing in an anonymous window will be protected from viruses\
\ launched from any web sites visited or files downloaded.\nA: Let's think step\
\ by step. Choice A is incorrect as it only describes network traffic, which an\
\ anonymous browser does not change. Choice B is correct as it correctly describes\
\ how an anonymous browser will prevent saving data on the user’s computer after\
\ the session is ended. Choice C is incorrect because an anonymous browser will\
\ not prevent logging in to email or social media accounts. Choice D is incorrect\
\ because an anonymous browser in itself performs no virus protection. The answer\
\ is (B).\n\nQ: In the program below, the initial value of X is 5 and the initial\
\ value of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\
\ DISPLAY (\"Hotel\")\n } ELSE {\n IF (Y > 0){\n DISPLAY (\"November\")\n }\
\ ELSE {\n DISPLAY (\"Yankee\")\n }\n }\n}\nWhat is displayed as a result of\
\ running the program?\n(A) Foxtrot (B) Hotel (C) November (D) Yankee\nA: Let's\
\ think step by step. Because X has the value 5, the first conditional IF (X < 0)\
\ is false, so we move to the first ELSE clause. Because X is 5 and Y is 10, the\
\ second conditional IF (X > Y) is false, so we move to the following ELSE clause.\
\ Since Y is 10, the conditional IF (Y > 0) is true, so the command DISPLAY (\"\
November\") is executed. The answer is (C).\n\nQ: What is the output of \"abc\"\
[::-1] in Python 3?\n(A) Error (B) abc (C) cba (D) c\nA: Let's think step by step.\
\ We know that the slicing operator [::-1] takes all of the elements in the string\
\ in reverse order, so we reverse the order of the string \"abc\", resulting in\
\ \"cba\". The answer is (C).\n\nQ: A list of numbers has n elements, indexed from\
\ 1 to n. The following algorithm is intended to display the number of elements\
\ in the list that have a value greater than 100. The algorithm uses the variables\
\ count and position. Steps 3 and 4 are missing.\n Step 1: Set count to 0 and position\
\ to 1.\n Step 2: If the value of the element at index position is greater than\
\ 100, increase the value of count by 1.\n Step 3: (missing step)\n Step 4: (missing\
\ step)\n Step 5: Display the value of count.\nWhich of the following could be used\
\ to replace steps 3 and 4 so that the algorithm works as intended?\n(A) Step 3:\
\ Increase the value of position by 1.\n Step 4: Repeat steps 2 and 3 until the\
\ value of count is greater than 100.\n(B) Step 3: Increase the value of position\
\ by 1.\n Step 4: Repeat steps 2 and 3 until the value of position is greater than\
\ n.\n(C) Step 3: Repeat step 2 until the value of count is greater than 100.\n\
\ Step 4: Increase the value of position by 1.\n(D) Step 3: Repeat step 2 until\
\ the value of position is greater than n.\n Step 4: Increase the value of count\
\ by 1.\nA: Let's think step by step. Choice A is incorrect, because its Step 4\
\ has an incorrect termination condition, stopping when count is greater than 100.\
\ We need to stop after inspecting all elements in the list. Choice B is correct\
\ because it correctly increments both count and position, and correctly repeats\
\ these steps and terminates when all elements in the list have been inspected.\
\ Choice C is incorrect because it incorrectly increments the variable count until\
\ its value is greater than 100, regardless of the elements in the list. Choice\
\ D is incorrect because its step 3 does not increment the value of position, so\
\ it will repeat forever. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_computer_science"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "high_school_european_history"
"description": "The following are multiple choice questions (with answers) about high\
\ school european history.\n\nQ: This question refers to the following information.\n\
Albeit the king's Majesty justly and rightfully is and ought to be the supreme head\
\ of the Church of England, and so is recognized by the clergy of this realm in\
\ their convocations, yet nevertheless, for corroboration and confirmation thereof,\
\ and for increase of virtue in Christ's religion within this realm of England,\
\ and to repress and extirpate all errors, heresies, and other enormities and abuses\
\ heretofore used in the same, be it enacted, by authority of this present Parliament,\
\ that the king, our sovereign lord, his heirs and successors, kings of this realm,\
\ shall be taken, accepted, and reputed the only supreme head in earth of the Church\
\ of England, called Anglicans Ecclesia; and shall have and enjoy, annexed and united\
\ to the imperial crown of this realm, as well the title and style thereof, as all\
\ honors, dignities, preeminences, jurisdictions, privileges, authorities, immunities,\
\ profits, and commodities to the said dignity of the supreme head of the same Church\
\ belonging and appertaining; and that our said sovereign lord, his heirs and successors,\
\ kings of this realm, shall have full power and authority from time to time to\
\ visit, repress, redress, record, order, correct, restrain, and amend all such\
\ errors, heresies, abuses, offenses, contempts, and enormities, whatsoever they\
\ be, which by any manner of spiritual authority or jurisdiction ought or may lawfully\
\ be reformed, repressed, ordered, redressed, corrected, restrained, or amended,\
\ most to the pleasure of Almighty God, the increase of virtue in Christ's religion,\
\ and for the conservation of the peace, unity, and tranquility of this realm; any\
\ usage, foreign land, foreign authority, prescription, or any other thing or things\
\ to the contrary hereof notwithstanding.\nEnglish Parliament, Act of Supremacy,\
\ 1534\nFrom the passage, one may infer that the English Parliament wished to argue\
\ that the Act of Supremacy would\n(A) give the English king a new position of authority\
\ (B) give the position of head of the Church of England to Henry VIII alone and\
\ exclude his heirs (C) establish Calvinism as the one true theology in England\
\ (D) end various forms of corruption plaguing the Church in England\nA: Let's think\
\ step by step. We refer to Wikipedia articles on european history for help. The\
\ Act of Supremacy states that it grants authority to the king \"to repress and\
\ extirpate all errors, heresies, and other enormities and abuses\", referring to\
\ the corruption in the Church of England. The answer is (D).\n\nQ: This question\
\ refers to the following information.\nRead the following excerpt.\nThe revolutionary\
\ seed had penetrated into every country and spread more or less. It was greatly\
\ developed under the régime of the military despotism of Bonaparte. His conquests\
\ displaced a number of laws, institutions, and customs; broke through bonds sacred\
\ among all nations, strong enough to resist time itself; which is more than can\
\ be said of certain benefits conferred by these innovators.\nThe monarchs will\
\ fulfil the duties imposed upon them by Him who, by entrusting them with power,\
\ has charged them to watch over the maintenance of justice, and the rights of all,\
\ to avoid the paths of error, and tread firmly in the way of truth. Placed beyond\
\ the passions which agitate society, it is in days of trial chiefly that they are\
\ called upon to despoil realities of their false appearances, and to show themselves\
\ as they are, fathers invested with the authority belonging by right to the heads\
\ of families, to prove that, in days of mourning, they know how to be just, wise,\
\ and therefore strong, and that they will not abandon the people whom they ought\
\ to govern to be the sport of factions, to error and its consequences, which must\
\ involve the loss of society.\nUnion between the monarchs is the basis of the policy\
\ which must now be followed to save society from total ruin. . . .\nLet them not\
\ confound concessions made to parties with the good they ought to do for their\
\ people, in modifying, according to their recognized needs, such branches of the\
\ administration as require it.\nLet them be just, but strong; beneficent, but strict.\n\
Let them maintain religious principles in all their purity, and not allow the faith\
\ to be attacked and morality interpreted according to the social contract or the\
\ visions of foolish sectarians.\nLet them suppress Secret Societies; that gangrene\
\ of society.\n—Klemens von Metternich, Political Confession of Faith, 1820\nWhich\
\ of the following was the greatest cause of the fears expressed by Metternich in\
\ the document above?\n(A) The ideas of personal liberty and nationalism conceived\
\ during the Enlightenment resulted in radical revolutions that could spread throughout\
\ Europe. (B) The conquest of Europe by Napoleon led to the creation of new factions\
\ and shifted the European balance of power. (C) The power of monarchs had grown\
\ to the point where it needed to be checked by other powers within each nation\
\ or domination of civilians would occur. (D) The rising and falling economic cycle\
\ of the newly emerging capitalist economy could lead to civilian unrest that must\
\ be suppressed.\nA: Let's think step by step. We refer to Wikipedia articles on\
\ european history for help. The fears of revolution in early 19th century Europe\
\ expressed by Klemens von Metternich, a conservative Austrian statesman, were a\
\ direct result of the age of Enlightenment, a period of European history where\
\ the absolute power of the monarchy was challenged with ideas of individual liberty\
\ and nationalism, leading to the French revolution and its effects all over Europe.\
\ The answer is (A).\n\nQ: This question refers to the following information.\n\
The excerpts below are from the Navigation Acts of 1651.\n[A]fter the first day\
\ of December, one thousand six hundred fifty and one, and from thence forwards,\
\ no goods or commodities whatsoever of the growth, production or manufacture of\
\ Asia, Africa or America, or of any part thereof; or of any islands belonging to\
\ them, or which are described or laid down in the usual maps or cards of those\
\ places, as well of the English plantations as others, shall be imported or brought\
\ into this Commonwealth of England, or into Ireland, or any other lands, islands,\
\ plantations, or territories to this Commonwealth belonging, or in their possession,\
\ in any other ship or ships, vessel or vessels whatsoever, but only in such as\
\ do truly and without fraud belong only to the people of this Commonwealth, or\
\ the plantations thereof, as the proprietors or right owners thereof; and whereof\
\ the master and mariners are also of the people of this Commonwealth, under the\
\ penalty of the forfeiture and loss of all the goods that shall be imported contrary\
\ to this act, , , ,\n[N]o goods or commodities of the growth, production, or manufacture\
\ of Europe, or of any part thereof, shall after the first day of December, one\
\ thousand six hundred fifty and one, be imported or brought into this Commonwealth\
\ of England, or any other lands or territories to this Commonwealth belonging,\
\ or in their possession, in any ship or ships, vessel or vessels whatsoever, but\
\ in such as do truly and without fraud belong only to the people of this Commonwealth,\
\ and in no other, except only such foreign ships and vessels as do truly and properly\
\ belong to the people of that country or place, of which the said goods are the\
\ growth, production or manufacture.\nWhich of the following best describes the\
\ outcome of the Navigation Acts of 1651?\n(A) They served as a catalyst for the\
\ growth of English shipping and overseas trade, but did little to limit the prospects\
\ of the Dutch in the seventeenth century. (B) They brought about almost immediate\
\ hardships for the Dutch economy as their dominance of overseas trade quickly ended.\
\ (C) They were rescinded during the restoration of the Stuarts as they sought normal\
\ diplomatic relations with the Dutch so not as to need Parliament's financial support\
\ for war. (D) They led to nearly a century of recurrent war between England and\
\ the Netherlands, which would not end until after American independence.\nA: Let's\
\ think step by step. We refer to Wikipedia articles on european history for help.\
\ The Navigation Acts of 1651 helped English shipping by restricting the ability\
\ of ships from other European countries, especially the Dutch, to transport goods\
\ from colonies in Asia and Africa into England. The answer is (A).\n\nQ: This question\
\ refers to the following information.\nIn Russia there was nothing going on well,\
\ and [Souvarine] was in despair over the news he had received. His old companions\
\ were all turning to the politicians; the famous Nihilists who made Europe tremble-sons\
\ of village priests, of the lower middle class, of tradesmen-could not rise above\
\ the idea of national liberation, and seemed to believe that the world would be\
\ delivered-when they had killed their despot&…\n\"Foolery! They'll never get out\
\ of it with their foolery.\"\nThen, lowering his voice still more, in a few bitter\
\ words he described his old dream of fraternity. He had renounced his rank and\
\ his fortune; he had gone among workmen, only in the hope of seeing at last the\
\ foundation of a new society of labour in common. All the sous in his pockets had\
\ long gone to the urchins of the settlement; he had been as tender as a brother\
\ with the colliers, smiling at their suspicion, winning them over by his quiet\
\ workmanlike ways and his dislike of chattering. But decidedly the fusion had not\
\ taken place.\nHis voice changed, his eyes grew bright, he fixed them on étienne,\
\ directly addressing him:\n\"Now, do you understand that? These hatworkers at Marseilles\
\ who have won the great lottery prize of a hundred thousand francs have gone off\
\ at once and invested it, declaring that they are going to live without doing anything!\
\ Yes, that is your idea, all of you French workmen; you want to unearth a treasure\
\ in order to devour it alone afterwards in some lazy, selfish corner. You may cry\
\ out as much as you like against the rich, you haven't got courage enough to give\
\ back to the poor the money that luck brings you. You will never be worthy of happiness\
\ as long as you own anything, and your hatred of the bourgeois proceeds solely\
\ from an angry desire to be bourgeois yourselves in their place.\"\némile Zola,\
\ French writer, Germinal, 1885\nThe passage displays the direct concern for the\
\ welfare of the working classes that was typically a part of which movement?\n\
(A) Capitalist (B) Scientific (C) Communist (D) Existentialist\nA: Let's think step\
\ by step. We refer to Wikipedia articles on european history for help. The modern\
\ Communist movement aims to establish a classless society based on communal ownership\
\ and distribution of property and means of production, thereby especially benefiting\
\ the working classes. The answer is (C).\n\nQ: This question refers to the following\
\ information.\nThe following excerpt is from a pamphlet.\nYou will do me the justice\
\ to remember, that I have always strenuously supported the Right of every man to\
\ his own opinion, however different that opinion might be to mine. He who denies\
\ to another this right, makes a slave of himself to his present opinion, because\
\ he precludes himself the right of changing it.\nThe most formidable weapon against\
\ errors of every kind is Reason. I have never used any other, and I trust I never\
\ shall.\nThe circumstance that has now taken place in France of the total abolition\
\ of the whole national order of priesthood, and of everything appertaining to compulsive\
\ systems of religion, and compulsive articles of faith, has not only precipitated\
\ my intention, but rendered a work of this kind exceedingly necessary, lest in\
\ the general wreck of superstition, of false systems of government, and false theology,\
\ we lose sight of morality, of humanity, and of the theology that is true.\nI believe\
\ in one God, and no more; and I hope for happiness beyond this life.\nI believe\
\ in the equality of man; and I believe that religious duties consist in doing justice,\
\ loving mercy, and endeavoring to make our fellow-creatures happy.\nI do not believe\
\ in the creed professed by the Jewish church, by the Roman church, by the Greek\
\ church, by the Turkish church, by the Protestant church, nor by any church that\
\ I know of. My own mind is my own church.\nAll national institutions of churches,\
\ whether Jewish, Christian or Turkish, appear to me no other than human inventions,\
\ set up to terrify and enslave mankind, and monopolize power and profit.\nI do\
\ not mean by this declaration to condemn those who believe otherwise; they have\
\ the same right to their belief as I have to mine.\n—Thomas Paine, The Age of Reason,\
\ 1794–1795\nWhich of the following Enlightenment philosophes designed a system\
\ of checks and balances for government to avoid abuses of power?\n(A) Jean Jacques\
\ Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\nA: Let's\
\ think step by step. We refer to Wikipedia articles on european history for help.\
\ Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\
\ against the monoplization of power and advocated for a system of checks and balances\
\ in government to prevent the rise of despotism. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_european_history"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml 0 → 100644
View file @ 6ac42518
"dataset_name": "high_school_geography"
"description": "The following are multiple choice questions (with answers) about high\
\ school geography.\n\nQ: Which one of the following items is an example of nonmaterial\
\ culture?\n(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).\n\
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
\ Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not\
\ physical objects. The answer is (C).\n\nQ: During the third stage of the demographic\
\ transition model, which of the following is true?\n(A) Birth rates increase and\
\ population growth rate is less rapid. (B) Birth rates decline and population growth\
\ rate is less rapid. (C) Birth rates increase and population growth rate increases.\
\ (D) Birth rates decrease and population growth rate increases.\nA: Let's think\
\ step by step. We refer to Wikipedia articles on geography for help. The demographic\
\ transition model models the five different stages of population growth as a country\
\ goes through economic development, where the third stage refers to a period of\
\ declining birth rates and lower population growth. The answer is (B).\n\nQ: The\
\ practice of hiring a foreign third-party service provider to run an operation\
\ is called\n(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.\n\
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
\ \"Offshoring\" literally means to move or base some of the activities or processes\
\ of a company to a foreign country. The answer is (B).\n\nQ: Which of the following\
\ statements is NOT accurate regarding the services provided by local governments\
\ in the United States?\n(A) Duplication of efforts occurs often. (B) Social problems\
\ of the central city spill over into the surrounding residential suburbs. (C) Inefficiency\
\ in providing services occurs often. (D) One neighborhood's efforts to reduce pollution\
\ are always supported by neighboring communities.\nA: Let's think step by step.\
\ We refer to Wikipedia articles on geography for help. There may be economic, social\
\ or political reasons for two neighboring communities and their local governments\
\ not agreeing to pollution reduction efforts initiated by one of them. The answer\
\ is (D).\n\nQ: The rate of natural increase of a population is found by subtracting\
\ the\n(A) crude death rate from the crude birth date. (B) crude birth rate from\
\ the crude death rate. (C) doubling time from the crude birth rate. (D) fertility\
\ rate from the crude death rate.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on geography for help. The difference between number of births and deaths\
\ gives the population increase at any given time. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_geography"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment