Skip to content

GitLab

Commit 2184b8de authored by lintangsutawika's avatar lintangsutawika
Browse files

Merge branch 'cont-metrics' of https://github.com/EleutherAI/lm-evaluation-harness into alt_worlds

parents b1ba4e71 1522009c
Hide whitespace changes
Inline Side-by-side
Showing with 711 additions and 914 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml
View file @ 2184b8de
dataset_name: clinical_knowledge "dataset_name": "clinical_knowledge"
description: 'The following are multiple choice questions (with answers) about clinical "description": "The following are multiple choice questions (with answers) about clinical\
knowledge. \ knowledge.\n\nQ: Glycolysis is the name given to the pathway involving the conversion\
\ of:\n(A) glycogen to glucose-1-phosphate. (B) glycogen or glucose to fructose.\
\ (C) glycogen or glucose to pyruvate or lactate. (D) glycogen or glucose to pyruvate\
Q: Glycolysis is the name given to the pathway involving the conversion of: \ or acetyl CoA.\nA: Let's think step by step. We refer to Wikipedia articles on\
\ clinical knowledge for help. Glycolysis is the name given to the pathway involving\
(A) glycogen to glucose-1-phosphate. (B) glycogen or glucose to fructose. (C) glycogen \ conversion of glycogen or glucose to pyruvate or lactate. The answer is (C).\n\
or glucose to pyruvate or lactate. (D) glycogen or glucose to pyruvate or acetyl \nQ: What is the difference between a male and a female catheter?\n(A) Male and\
CoA. \ female catheters are different colours. (B) Male catheters are longer than female\
\ catheters. (C) Male catheters are bigger than female catheters. (D) Female catheters\
A: Let''s think step by step. We refer to Wikipedia articles on clinical knowledge \ are longer than male catheters.\nA: Let's think step by step. We refer to Wikipedia\
for help. Glycolysis is the name given to the pathway involving conversion of glycogen \ articles on clinical knowledge for help. The difference between a male and female\
or glucose to pyruvate or lactate. The answer is (C). \ catheter is that male catheters tend to be longer than female catheters. The answer\
\ is (B).\n\nQ: How many attempts should you make to cannulate a patient before\
\ passing the job on to a senior colleague, according to the medical knowledge of\
Q: What is the difference between a male and a female catheter? \ 2020?\n(A) 4 (B) 3 (C) 2 (D) 1\nA: Let's think step by step. We refer to Wikipedia\
\ articles on clinical knowledge for help. According to the medical protocol as\
(A) Male and female catheters are different colours. (B) Male catheters are longer \ of 2020, you should make two attempts to cannulate a patient before passing the\
than female catheters. (C) Male catheters are bigger than female catheters. (D) \ job on to a more-senior practitioner. The answer is (C).\n\nQ: In the assessment\
Female catheters are longer than male catheters. \ of the hand function which of the following is true?\n(A) Abduction of the thumb\
\ is supplied by spinal root T2 (B) Opposition of the thumb by opponens policis\
A: Let''s think step by step. We refer to Wikipedia articles on clinical knowledge \ is supplied by spinal root T1 (C) Finger adduction is supplied by the median nerve\
for help. The difference between a male and female catheter is that male catheters \ (D) Finger abduction is mediated by the palmar interossei\nA: Let's think step\
tend to be longer than female catheters. The answer is (B). \ by step. We refer to Wikipedia articles on clinical knowledge for help. Of all\
\ the options, it is only true that the opposition of the thumb by opponens pollicis\
\ is supplied by spinal root T1. The answer is (B).\n\nQ: The energy for all forms\
Q: How many attempts should you make to cannulate a patient before passing the job \ of muscle contraction is provided by:\n(A) ATP. (B) ADP. (C) phosphocreatine.\
on to a senior colleague, according to the medical knowledge of 2020? \ (D) oxidative phosphorylation.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on clinical knowledge for help. The energy for muscular contraction is\
(A) 4 (B) 3 (C) 2 (D) 1 \ provided by ATP (adenosine triphosphate), which is the powerhouse of the cell.\
\ The answer is (A)."
A: Let''s think step by step. We refer to Wikipedia articles on clinical knowledge "group": "mmlu_flan_cot_fewshot_other"
for help. According to the medical protocol as of 2020, you should make two attempts "include": "_mmlu_flan_cot_fewshot_template_yaml"
to cannulate a patient before passing the job on to a more-senior practitioner. "task": "mmlu_flan_cot_fewshot_clinical_knowledge"
The answer is (C).
Q: In the assessment of the hand function which of the following is true?
(A) Abduction of the thumb is supplied by spinal root T2 (B) Opposition of the thumb
by opponens policis is supplied by spinal root T1 (C) Finger adduction is supplied
by the median nerve (D) Finger abduction is mediated by the palmar interossei
A: Let''s think step by step. We refer to Wikipedia articles on clinical knowledge
for help. Of all the options, it is only true that the opposition of the thumb by
opponens pollicis is supplied by spinal root T1. The answer is (B).
Q: The energy for all forms of muscle contraction is provided by:
(A) ATP. (B) ADP. (C) phosphocreatine. (D) oxidative phosphorylation.
A: Let''s think step by step. We refer to Wikipedia articles on clinical knowledge
for help. The energy for muscular contraction is provided by ATP (adenosine triphosphate),
which is the powerhouse of the cell. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_clinical_knowledge
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml
View file @ 2184b8de
dataset_name: college_biology "dataset_name": "college_biology"
description: "The following are multiple choice questions (with answers) about college\ "description": "The following are multiple choice questions (with answers) about college\
\ biology.\n\nQ: Which of the following represents an accurate statement concerning\ \ biology.\n\nQ: Which of the following represents an accurate statement concerning\
\ arthropods?\n(A) They possess an exoskeleton composed primarily of peptidoglycan.\ \ arthropods?\n(A) They possess an exoskeleton composed primarily of peptidoglycan.\
\ (B) They possess an open circulatory system with a dorsal heart. (C) They are\ \ (B) They possess an open circulatory system with a dorsal heart. (C) They are\
...@@ -19,7 +19,7 @@ description: "The following are multiple choice questions (with answers) about c ...@@ -19,7 +19,7 @@ description: "The following are multiple choice questions (with answers) about c
\ Law, $p^2 + 2 p q + q^2 = 1$, and $p + q = 1$ where $p$ is the frequency of the\ \ Law, $p^2 + 2 p q + q^2 = 1$, and $p + q = 1$ where $p$ is the frequency of the\
\ dominant allele, $q$ is the frequency of the recessive allele, and $p^2$, $q^2$,\ \ dominant allele, $q$ is the frequency of the recessive allele, and $p^2$, $q^2$,\
\ and $2pq$ are the frequencies of dominant homozygous, recessive homozygous, and\ \ and $2pq$ are the frequencies of dominant homozygous, recessive homozygous, and\
\ heterozygous individuals, respectively. \u200BThe frequency of the recessive allele\ \ heterozygous individuals, respectively. ​The frequency of the recessive allele\
\ (q) is $\\sqrt{\frac{1}{400}} = 0.05$. We have $p = 1 - q = 0.95$. The frequency\ \ (q) is $\\sqrt{\frac{1}{400}} = 0.05$. We have $p = 1 - q = 0.95$. The frequency\
\ of heterozygous individuals is $2pq = 2 \\cdot 0.05 \\cdot 0.95 = 0.095$. The\ \ of heterozygous individuals is $2pq = 2 \\cdot 0.05 \\cdot 0.95 = 0.095$. The\
\ number of heterozygous individuals is equal to the frequency of heterozygous individuals\ \ number of heterozygous individuals is equal to the frequency of heterozygous individuals\
...@@ -56,5 +56,6 @@ description: "The following are multiple choice questions (with answers) about c ...@@ -56,5 +56,6 @@ description: "The following are multiple choice questions (with answers) about c
\ the human and bird forearms, which rules out (D). Humans and birds do belong to\ \ the human and bird forearms, which rules out (D). Humans and birds do belong to\
\ the same clade - a group of organisms composed of a common ancestor. The answer\ \ the same clade - a group of organisms composed of a common ancestor. The answer\
\ is (C)." \ is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_college_biology "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_biology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml
View file @ 2184b8de
dataset_name: college_chemistry "dataset_name": "college_chemistry"
description: "The following are multiple choice questions (with answers) about college\ "description": "The following are multiple choice questions (with answers) about college\
\ chemistry.\n\nQ: 3 Cl\u2212(aq) + 4 CrO_4^2\u2212(aq) + 23 H+(aq) \u2192 3 HClO2(aq)\ \ chemistry.\n\nQ: 3 Cl−(aq) + 4 CrO_4^2−(aq) + 23 H+(aq) → 3 HClO2(aq) + 4 Cr3+(aq)\
\ + 4 Cr3+(aq) + 10 H2O(l). In the reaction shown above, Cl\u2212(aq) behaves as\n\ \ + 10 H2O(l). In the reaction shown above, Cl−(aq) behaves as\n(A) an acid (B)\
(A) an acid (B) a base (C) a catalyst (D) a reducing agent\nA: Let's think step\ \ a base (C) a catalyst (D) a reducing agent\nA: Let's think step by step. A molecule\
\ by step. A molecule that behaves as a base accepts an H+ ion (or proton) from\ \ that behaves as a base accepts an H+ ion (or proton) from another molecule, whereas\
\ another molecule, whereas a molecule that behaves as an acid donates an H+ ion\ \ a molecule that behaves as an acid donates an H+ ion (or proton) to another molecule.\
\ (or proton) to another molecule. Neither of these is the case for Cl in this reaction,\ \ Neither of these is the case for Cl in this reaction, which rules out (A) and\
\ which rules out (A) and (B). A catalyst is a substance that only accelerates a\ \ (B). A catalyst is a substance that only accelerates a reaction without itself\
\ reaction without itself undergoing chemical change, which is not the case here.\ \ undergoing chemical change, which is not the case here. This rules out (C). Instead,\
\ This rules out (C). Instead, the $Cl^{-} molecules carry a negative charge, which\ \ the $Cl^{-} molecules carry a negative charge, which they donate in the reaction\
\ they donate in the reaction to form 3 HClO2. This is the behavior of a reducing\ \ to form 3 HClO2. This is the behavior of a reducing agent, or (D). The answer\
\ agent, or (D). The answer is (D).\n\nQ: Which of the following statements about\ \ is (D).\n\nQ: Which of the following statements about the lanthanide elements\
\ the lanthanide elements is NOT true?\n(A) The most common oxidation state for\ \ is NOT true?\n(A) The most common oxidation state for the lanthanide elements\
\ the lanthanide elements is +3. (B) Lanthanide complexes often have high coordination\ \ is +3. (B) Lanthanide complexes often have high coordination numbers (> 6). (C)\
\ numbers (> 6). (C) All of the lanthanide elements react with aqueous acid to liberate\ \ All of the lanthanide elements react with aqueous acid to liberate hydrogen. (D)\
\ hydrogen. (D) The atomic radii of the lanthanide elements increase across the\ \ The atomic radii of the lanthanide elements increase across the period from La\
\ period from La to Lu.\nA: Let's think step by step. The atomic radii of the lanthanide\ \ to Lu.\nA: Let's think step by step. The atomic radii of the lanthanide elements\
\ elements in fact decrease across the period from La to Lu. Options (A), (B), and\ \ in fact decrease across the period from La to Lu. Options (A), (B), and (C) are\
\ (C) are all true. This means that only (D) is NOT true. The answer is (D).\n\n\ \ all true. This means that only (D) is NOT true. The answer is (D).\n\nQ: Which\
Q: Which of the following lists the hydrides of group-14 elements in order of thermal\ \ of the following lists the hydrides of group-14 elements in order of thermal stability,\
\ stability, from lowest to highest?\n(A) PbH4 < SnH4 < GeH4 < SiH4 < CH4 (B) PbH4\ \ from lowest to highest?\n(A) PbH4 < SnH4 < GeH4 < SiH4 < CH4 (B) PbH4 < SnH4 <\
\ < SnH4 < CH4 < GeH4 < SiH4 (C) CH4 < SiH4 < GeH4 < SnH4 < PbH4 (D) CH4 < PbH4\ \ CH4 < GeH4 < SiH4 (C) CH4 < SiH4 < GeH4 < SnH4 < PbH4 (D) CH4 < PbH4 < GeH4 <\
\ < GeH4 < SnH4 < SiH4\nA: Let's think step by step. The thermal stability of group-14\ \ SnH4 < SiH4\nA: Let's think step by step. The thermal stability of group-14 hydrides\
\ hydrides decreases as we move from the top of group 14 to the bottom. The order\ \ decreases as we move from the top of group 14 to the bottom. The order of elements\
\ of elements in the group from top to bottom is C, Si, Ge, Sn, Pb. Therefore in\ \ in the group from top to bottom is C, Si, Ge, Sn, Pb. Therefore in order of increasing\
\ order of increasing thermal stability we have PbH4, SnH4, GeH4, SiH4, and CH4,\ \ thermal stability we have PbH4, SnH4, GeH4, SiH4, and CH4, or answer (A). The\
\ or answer (A). The answer is (A).\n\nQ: Predict the number of lines in the EPR\ \ answer is (A).\n\nQ: Predict the number of lines in the EPR spectrum of a solution\
\ spectrum of a solution of 13C-labelled methyl radical (13CH3\u2022), assuming\ \ of 13C-labelled methyl radical (13CH3•), assuming the lines do not overlap.\n\
\ the lines do not overlap.\n(A) 4 (B) 3 (C) 6 (D) 24 (E) 8\nA: Let's think step\ (A) 4 (B) 3 (C) 6 (D) 24 (E) 8\nA: Let's think step by step. The electron paramagnetic\
\ by step. The electron paramagnetic resonance spectrum will be split by two forms\ \ resonance spectrum will be split by two forms of interactions. The first is the\
\ of interactions. The first is the hyperfine interaction with the 13C (nuclear\ \ hyperfine interaction with the 13C (nuclear spin $I = \nrac{1}{2}$) which will\
\ spin $I = \nrac{1}{2}$) which will split the spectrum into 2 lines. This will\ \ split the spectrum into 2 lines. This will be further split into 4 lines by the\
\ be further split into 4 lines by the interaction with three equivalent 1H nuclei.\ \ interaction with three equivalent 1H nuclei. The total number of lines is therefore\
\ The total number of lines is therefore $2 \\cdot 4 = 8$. The answer is (E)." \ $2 \\cdot 4 = 8$. The answer is (E)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_college_chemistry "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_chemistry"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml
View file @ 2184b8de
dataset_name: college_computer_science "dataset_name": "college_computer_science"
description: 'The following are multiple choice questions (with answers) about college "description": "The following are multiple choice questions (with answers) about college\
computer science. \ computer science.\n\nQ: Which of the following regular expressions is equivalent\
\ to (describes the same set of strings as) (a* + b)*(c + d)?\n(A) a*(c + d)+ b(c\
\ + d)\n(B) a*(c + d)* + b(c + d)*\n(C) a*(c + d)+ b*(c + d)\n(D) (a + b)*c +(a\
Q: Which of the following regular expressions is equivalent to (describes the same \ + b)*d\nA: Let's think step by step. We know that:\n1. (X* + Y)* = (X + Y)*\n\
set of strings as) (a* + b)*(c + d)? 2. X(Y + Z)? = XY + XZ\nUsing equation 1 we can rewrite (a* + b)*(c + d)? as:\n\
3. (a + b)*(c + d)?\nUsing equation 2 we can rewrite equation 3 as:\n(a + b)*c +\
(A) a*(c + d)+ b(c + d) \ (a + b)*d The answer is (D).\n\nQ: The Singleton design pattern is used to guarantee\
\ that only a single instance of a class may be instantiated. Which of the following\
(B) a*(c + d)* + b(c + d)* \ is (are) true of this design pattern?\nI. The Singleton class has a static factory\
\ method to provide its instance.\nII. The Singleton class can be a subclass of\
(C) a*(c + d)+ b*(c + d) \ another class.\nIII. The Singleton class has a private constructor.\n(A) I only\n\
(B) II only\n(C) III only\n(D) I, II, and III\nA: Let's think step by step. Statement\
(D) (a + b)*c +(a + b)*d \ I is a correct statement about a Singleton, because a Singleton restricts instantiation\
\ to a single, static method. Statement II is also correct, because there is no\
A: Let''s think step by step. We know that: \ inherent restriction regarding the inheritance of a Singleton. Statement III is\
\ also correct, because a Singletons must be instantiated only once, so its constructor\
1. (X* + Y)* = (X + Y)* \ is made private to prevent any construction except via its static factory method.\n\
Given these facts, statements I, II, and III are all correct. The answer is (D).\n\
2. X(Y + Z)? = XY + XZ \nQ: A certain pipelined RISC machine has 8 general-purpose registers R0, R1, .\
\ . . , R7 and supports the following operations:\nADD Rs1, Rs2, Rd (Add Rs1 to\
Using equation 1 we can rewrite (a* + b)*(c + d)? as: \ Rs2 and put the sum in Rd)\nMUL Rs1, Rs2, Rd (Multiply Rs1 by Rs2 and put the\
\ product in Rd)\nAn operation normally takes one cycle; however, an operation takes\
3. (a + b)*(c + d)? \ two cycles if it produces a result required by the immediately following operation\
\ in an operation sequence.\nConsider the expression AB + ABC + BC, where variables\
Using equation 2 we can rewrite equation 3 as: \ A, B, C are located in registers R0, R1, R2. If the contents of these three registers\
\ must not be modified, what is the minimum number of clock cycles required for\
(a + b)*c + (a + b)*d The answer is (D). \ an operation sequence that computes the value of AB + ABC + BC?\n(A) 5 (B) 6 (C)\
\ 7 (D) 8\nA: Let's think step by step. First, we are given that A is in R0, B is\
\ in R1, and C is in R2.\nNext, we can see that we must compute three multiplies\
Q: The Singleton design pattern is used to guarantee that only a single instance \ (AB, BC, and ABC) and two adds (AB + ABC, (AB + ABC) + BC) to compute our final\
of a class may be instantiated. Which of the following is (are) true of this design \ answer, resulting in a minimum of five clock cycles.\nNext, we can see that there\
pattern? \ is no way to avoid at least one pipeline stall when computing our final answer,\
\ because to compute our final sum we must wait at least one cycle for the results\
I. The Singleton class has a static factory method to provide its instance. \ from the previous stage to be ready. Thus, our minimum number of cycles must be\
\ 6.\nWe can verify that we can create a solution that requires only six cycles\
II. The Singleton class can be a subclass of another class. \ as follows:\ncompute AB: MUL R0, R1, R3\ncompute BC: MUL R1, R2, R4\ncompute ABC:\
\ MUL R3, R4, R5\ncompute AB + BC: ADD R3, R4, R6\nSTALL\ncompute AB + ABC + BC:\
III. The Singleton class has a private constructor. \ ADD R5, R6, R7\nSo there are 6 cycles. The answer is (B).\n\nQ: A compiler generates\
\ code for the following assignment statement.\nG := (A + B) * C - (D + E) * F\n\
(A) I only The target machine has a single accumulator and a single-address instruction set\
\ consisting of instructions load, store, add, subtract, and multiply. For the arithmetic\
(B) II only \ operations, the left operand is taken from the accumulator and the result appears\
\ in the accumulator. The smallest possible number of instructions in the resulting\
(C) III only \ code is\n(A) 5 (B) 6 (C) 7 (D) 9\nA: Let's think step by step. We can compute\
\ the final answer with the following sequence of operations:\n1. LOAD D (accumulator\
(D) I, II, and III \ = D)\n2. ADD E (accumulator = D+E)\n3. MUL F (accumulator = (D+E)*F)\n4. STORE\
\ X (X = (D+E)*F)\n5. LOAD A (accumulator = A)\n6. ADD B (accumulator = A+B)\n\
A: Let''s think step by step. Statement I is a correct statement about a Singleton, 7. MUL C (accumulator = (A+B)*C)\n8. SUB X (accumulator = (A+B)*C - (D+E)*F)\n\
because a Singleton restricts instantiation to a single, static method. Statement 9. STORE G (G = (A+B)*C - (D+E)*F)\nThis sequence takes 9 instructions. The answer\
II is also correct, because there is no inherent restriction regarding the inheritance \ is (D).\n\nQ: Consider a computer design in which multiple processors, each with\
of a Singleton. Statement III is also correct, because a Singletons must be instantiated \ a private cache memory, share global memory using a single bus. This bus is the\
only once, so its constructor is made private to prevent any construction except \ critical system resource. Each processor can execute one instruction every 500\
via its static factory method. \ nanoseconds as long as memory references are satisfied by its local cache. When\
\ a cache miss occurs, the processor is delayed for an additional 2,000 nanoseconds.\
Given these facts, statements I, II, and III are all correct. The answer is (D). \ During half of this additional delay, the bus is dedicated to serving the cache\
\ miss. During the other half, the processor cannot continue, but the bus is free\
\ to service requests from other processors. On average, each instruction requires\
Q: A certain pipelined RISC machine has 8 general-purpose registers R0, R1, . . \ 2 memory references. On average, cache misses occur on 1 percent of references.\
. , R7 and supports the following operations: \ What proportion of the capacity of the bus would a single processor consume, ignoring\
\ delays due to competition from other processors?\n(A) 1/50 (B) 1/27 (C) 1/25 (D)\
ADD Rs1, Rs2, Rd (Add Rs1 to Rs2 and put the sum in Rd) \ 2/27\nA: Let's think step by step. We know that each instruction requires two\
\ memory references per instruction, and that there is an average cache miss rate\
MUL Rs1, Rs2, Rd (Multiply Rs1 by Rs2 and put the product in Rd) \ of one percent.\nThus a given processor has:\n(1 cache miss / 100 references)\
\ * (2 references / instruction) =\n(2 cache misses / 100 instructions), so:\nmisses_per_instruction\
An operation normally takes one cycle; however, an operation takes two cycles if \ = 1 cache miss / 50 instructions.\nNext, we know that each instruction requires\
it produces a result required by the immediately following operation in an operation \ 500 nanoseconds when there is no cache miss, and 500 + 2000 = 2500 nanoseconds\
sequence. \ when there is a cache miss. Thus:\n50 instructions / (49 * 500) + (1 * 2500) nanoseconds,\
\ so:\ninstructions_per_ns = 50 instructions / 27000 nanoseconds.\nNow, we know\
Consider the expression AB + ABC + BC, where variables A, B, C are located in registers \ that each cache miss locks the bus for half of the 2000 nanosecond cache miss\
R0, R1, R2. If the contents of these three registers must not be modified, what \ delay, or 1000 nanoseconds, so:\nlock_ns_per_miss = 1000 nanoseconds / cache miss.\n\
is the minimum number of clock cycles required for an operation sequence that computes Thus we can see that on average a single processor will lock the bus for:\nlock_ns_per_miss\
the value of AB + ABC + BC? \ * misses_per_instruction * instructions_per_ns =\n(1000 nanoseconds / cache miss)\
\ * (1 cache miss / 50 instructions) * (50 instructions / 27000 nanoseconds) = 1000\
(A) 5 (B) 6 (C) 7 (D) 8 \ * (1/50) * (50/27000) = 1000/27000 = 1/27. The answer is (B)."
"group": "mmlu_flan_cot_fewshot_stem"
A: Let''s think step by step. First, we are given that A is in R0, B is in R1, and "include": "_mmlu_flan_cot_fewshot_template_yaml"
C is in R2. "task": "mmlu_flan_cot_fewshot_college_computer_science"
Next, we can see that we must compute three multiplies (AB, BC, and ABC) and two
adds (AB + ABC, (AB + ABC) + BC) to compute our final answer, resulting in a minimum
of five clock cycles.
Next, we can see that there is no way to avoid at least one pipeline stall when
computing our final answer, because to compute our final sum we must wait at least
one cycle for the results from the previous stage to be ready. Thus, our minimum
number of cycles must be 6.
We can verify that we can create a solution that requires only six cycles as follows:
compute AB: MUL R0, R1, R3
compute BC: MUL R1, R2, R4
compute ABC: MUL R3, R4, R5
compute AB + BC: ADD R3, R4, R6
STALL
compute AB + ABC + BC: ADD R5, R6, R7
So there are 6 cycles. The answer is (B).
Q: A compiler generates code for the following assignment statement.
G := (A + B) * C - (D + E) * F
The target machine has a single accumulator and a single-address instruction set
consisting of instructions load, store, add, subtract, and multiply. For the arithmetic
operations, the left operand is taken from the accumulator and the result appears
in the accumulator. The smallest possible number of instructions in the resulting
code is
(A) 5 (B) 6 (C) 7 (D) 9
A: Let''s think step by step. We can compute the final answer with the following
sequence of operations:
1. LOAD D (accumulator = D)
2. ADD E (accumulator = D+E)
3. MUL F (accumulator = (D+E)*F)
4. STORE X (X = (D+E)*F)
5. LOAD A (accumulator = A)
6. ADD B (accumulator = A+B)
7. MUL C (accumulator = (A+B)*C)
8. SUB X (accumulator = (A+B)*C - (D+E)*F)
9. STORE G (G = (A+B)*C - (D+E)*F)
This sequence takes 9 instructions. The answer is (D).
Q: Consider a computer design in which multiple processors, each with a private
cache memory, share global memory using a single bus. This bus is the critical system
resource. Each processor can execute one instruction every 500 nanoseconds as long
as memory references are satisfied by its local cache. When a cache miss occurs,
the processor is delayed for an additional 2,000 nanoseconds. During half of this
additional delay, the bus is dedicated to serving the cache miss. During the other
half, the processor cannot continue, but the bus is free to service requests from
other processors. On average, each instruction requires 2 memory references. On
average, cache misses occur on 1 percent of references. What proportion of the capacity
of the bus would a single processor consume, ignoring delays due to competition
from other processors?
(A) 1/50 (B) 1/27 (C) 1/25 (D) 2/27
A: Let''s think step by step. We know that each instruction requires two memory
references per instruction, and that there is an average cache miss rate of one
percent.
Thus a given processor has:
(1 cache miss / 100 references) * (2 references / instruction) =
(2 cache misses / 100 instructions), so:
misses_per_instruction = 1 cache miss / 50 instructions.
Next, we know that each instruction requires 500 nanoseconds when there is no cache
miss, and 500 + 2000 = 2500 nanoseconds when there is a cache miss. Thus:
50 instructions / (49 * 500) + (1 * 2500) nanoseconds, so:
instructions_per_ns = 50 instructions / 27000 nanoseconds.
Now, we know that each cache miss locks the bus for half of the 2000 nanosecond
cache miss delay, or 1000 nanoseconds, so:
lock_ns_per_miss = 1000 nanoseconds / cache miss.
Thus we can see that on average a single processor will lock the bus for:
lock_ns_per_miss * misses_per_instruction * instructions_per_ns =
(1000 nanoseconds / cache miss) * (1 cache miss / 50 instructions) * (50 instructions
/ 27000 nanoseconds) = 1000 * (1/50) * (50/27000) = 1000/27000 = 1/27. The answer
is (B).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_college_computer_science
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml
View file @ 2184b8de
dataset_name: college_mathematics "dataset_name": "college_mathematics"
description: "The following are multiple choice questions (with answers) about college\ "description": "The following are multiple choice questions (with answers) about college\
\ mathematics.\n\nQ: Let V be the set of all real polynomials p(x). Let transformations\ \ mathematics.\n\nQ: Let V be the set of all real polynomials p(x). Let transformations\
\ T, S be defined on V by T:p(x) -> xp(x) and S:p(x) -> p'(x) = d/dx p(x), and interpret\ \ T, S be defined on V by T:p(x) -> xp(x) and S:p(x) -> p'(x) = d/dx p(x), and interpret\
\ (ST)(p(x)) as S(T(p(x))). Which of the following is true?\n(A) ST = 0 (B) ST =\ \ (ST)(p(x)) as S(T(p(x))). Which of the following is true?\n(A) ST = 0 (B) ST =\
\ T (C) ST = TS (D) ST - TS is the identity map of V onto itself.\nA: Let's think\ \ T (C) ST = TS (D) ST - TS is the identity map of V onto itself.\nA: Let's think\
\ step by step. For a given polynomial $p$ we have\n\\[ST(p) = (xp(x))\u2019 = p(x)\ \ step by step. For a given polynomial $p$ we have\n\\[ST(p) = (xp(x))’ = p(x) +\
\ + xp\u2019(x)\\]\nand\n\\[TS(p) = xp\u2019(x).\\]\nHence \\[ST(p) - TS(p) = p(x)\ \ xp’(x)\\]\nand\n\\[TS(p) = xp’(x).\\]\nHence \\[ST(p) - TS(p) = p(x) + xp’(x)\
\ + xp\u2019(x) - xp\u2019(x).\\] The answer is (D).\n\nQ: Suppose that f(1 + x)\ \ - xp’(x).\\] The answer is (D).\n\nQ: Suppose that f(1 + x) = f(x) for all real\
\ = f(x) for all real x. If f is a polynomial and f(5) = 11, then f(15/2)\n(A) -11\ \ x. If f is a polynomial and f(5) = 11, then f(15/2)\n(A) -11 (B) 0 (C) 11 (D)\
\ (B) 0 (C) 11 (D) 33/2\nA: Let's think step by step. The only polynomial so that\ \ 33/2\nA: Let's think step by step. The only polynomial so that $f(1 + x) = f(x)$\
\ $f(1 + x) = f(x)$ is a constant polynomial. Hence $f(5) = 11 = f(15/2)$. The answer\ \ is a constant polynomial. Hence $f(5) = 11 = f(15/2)$. The answer is (C).\n\n\
\ is (C).\n\nQ: Let A be a real 2x2 matrix. Which of the following statements must\ Q: Let A be a real 2x2 matrix. Which of the following statements must be true?\n\
\ be true?\nI. All of the entries of A^2 are nonnegative.\nII. The determinant of\ I. All of the entries of A^2 are nonnegative.\nII. The determinant of A^2 is nonnegative.\n\
\ A^2 is nonnegative.\nIII. If A has two distinct eigenvalues, then A^2 has two\ III. If A has two distinct eigenvalues, then A^2 has two distinct eigenvalues.\n\
\ distinct eigenvalues.\n(A) I only (B) II only (C) III only (D) II and III only\n\ (A) I only (B) II only (C) III only (D) II and III only\nA: Let's think step by\
A: Let's think step by step. We have \\[ det(A^2) = (det(A))^2 \\geq 0,\\] hence\ \ step. We have \\[ det(A^2) = (det(A))^2 \\geq 0,\\] hence II holds.\nIII is false:\
\ II holds.\nIII is false: as a counterexample take a diagonal matrix with -1 and\ \ as a counterexample take a diagonal matrix with -1 and 1 on the diagonal. Then\
\ 1 on the diagonal. Then $A^2$ is the identity matrix. The answer is (B).\n\nQ:\ \ $A^2$ is the identity matrix. The answer is (B).\n\nQ: Let A be the set of all\
\ Let A be the set of all ordered pairs of integers (m, n) such that 7m + 12n =\ \ ordered pairs of integers (m, n) such that 7m + 12n = 22. What is the greatest\
\ 22. What is the greatest negative number in the set B = {m + n : (m, n) \\in A}?\n\ \ negative number in the set B = {m + n : (m, n) \\in A}?\n(A) -5 (B) -4 (C) -3\
(A) -5 (B) -4 (C) -3 (D) -2\nA: Let's think step by step. We have 12n = 22 - 7m\ \ (D) -2\nA: Let's think step by step. We have 12n = 22 - 7m and one of the solutions\
\ and one of the solutions is $m = -2$, $n = 3$. Then $m + n = 1$, hence we need\ \ is $m = -2$, $n = 3$. Then $m + n = 1$, hence we need to look for smaller $m$\
\ to look for smaller $m$ in order to make $m + n$ negative. The next solution is\ \ in order to make $m + n$ negative. The next solution is $m = -14$ and $n = 10$.\
\ $m = -14$ and $n = 10$. For smaller $m$ we have $m + n$ smaller than $-4$. The\ \ For smaller $m$ we have $m + n$ smaller than $-4$. The answer is (B).\n\nQ: A\
\ answer is (B).\n\nQ: A tank initially contains a salt solution of 3 grams of salt\ \ tank initially contains a salt solution of 3 grams of salt dissolved in 100 liters\
\ dissolved in 100 liters of water. A salt solution containing 0.02 grams of salt\ \ of water. A salt solution containing 0.02 grams of salt per liter of water is\
\ per liter of water is sprayed into the tank at a rate of 4 liters per minute.\ \ sprayed into the tank at a rate of 4 liters per minute. The sprayed solution is\
\ The sprayed solution is continually mixed with the salt solution in the tank,\ \ continually mixed with the salt solution in the tank, and the mixture flows out\
\ and the mixture flows out of the tank at a rate of 4 liters per minute. If the\ \ of the tank at a rate of 4 liters per minute. If the mixing is instantaneous,\
\ mixing is instantaneous, how many grams of salt are in the tank after 100 minutes\ \ how many grams of salt are in the tank after 100 minutes have elapsed?\n(A) 2\
\ have elapsed?\n(A) 2 (B) 2 - e^-2 (C) 2 + e^-2 (D) 2 + e^-4\nA: Let's think step\ \ (B) 2 - e^-2 (C) 2 + e^-2 (D) 2 + e^-4\nA: Let's think step by step. For all $t\
\ by step. For all $t \\in \\mathbb{R}$, let $s(t)$ denote the number grams of salt\ \ \\in \\mathbb{R}$, let $s(t)$ denote the number grams of salt in the tank at the\
\ in the tank at the $t$ minute mark. Then $s(0) = 3$.\nWe use $s$ and $s(t)$ interchangeably.\ \ $t$ minute mark. Then $s(0) = 3$.\nWe use $s$ and $s(t)$ interchangeably. We also\
\ We also use $s^{\\prime}$ and $s^{\\prime}(t)$ interchangeably. The solution sprayed\ \ use $s^{\\prime}$ and $s^{\\prime}(t)$ interchangeably. The solution sprayed into\
\ into the tank adds $(0.02) 4=2 / 25$ grams of salt per minute. There are always\ \ the tank adds $(0.02) 4=2 / 25$ grams of salt per minute. There are always 100\
\ 100 liters of liquid in the tank, containing $s$ grams of salt. So the density\ \ liters of liquid in the tank, containing $s$ grams of salt. So the density of\
\ of salt in the tank is $s / 100$ grams per liter. The flow of water out of the\ \ salt in the tank is $s / 100$ grams per liter. The flow of water out of the tank\
\ tank therefore subtracts $4(s / 100)=s / 25$ grams of salt per minute. Then, for\ \ therefore subtracts $4(s / 100)=s / 25$ grams of salt per minute. Then, for all\
\ all $t \\in \\mathbb{R}$, we have $s^{\\prime}(t)=(2 / 25)-(s / 25)=(2-s) / 25$,\ \ $t \\in \\mathbb{R}$, we have $s^{\\prime}(t)=(2 / 25)-(s / 25)=(2-s) / 25$, and\
\ and so $[s(t)=2] \\Rightarrow\\left[s^{\\prime}(t)=0\right]$. For all $t \\in\ \ so $[s(t)=2] \\Rightarrow\\left[s^{\\prime}(t)=0\right]$. For all $t \\in \\mathbb{R}$,\n\
\ \\mathbb{R}$,\n$$\n\frac{d}{d t}[\\ln (s-2)]=\frac{s^{\\prime}}{s-2}=\frac{-1}{25}=\f\ $$\n\frac{d}{d t}[\\ln (s-2)]=\frac{s^{\\prime}}{s-2}=\frac{-1}{25}=\frac{d}{d t}\\\
rac{d}{d t}\\left[-\frac{t}{25}\right] .\n$$\nChoose $C \\in \\mathbb{R}$ such that,\ left[-\frac{t}{25}\right] .\n$$\nChoose $C \\in \\mathbb{R}$ such that, for all\
\ for all $t \\in \\mathbb{R}, \\ln ((s(t)-2))=-[t / 25]+C$. Let $K:=e^{C}$. Then,\ \ $t \\in \\mathbb{R}, \\ln ((s(t)-2))=-[t / 25]+C$. Let $K:=e^{C}$. Then, for all\
\ for all $t \\in \\mathbb{R}$, we have $(s(t))-2=K e^{-t / 25}$, and so $s(t)=2+K\ \ $t \\in \\mathbb{R}$, we have $(s(t))-2=K e^{-t / 25}$, and so $s(t)=2+K e^{-t\
\ e^{-t / 25}$. Then $3=s(0)=2+K e^{0}=2+K$, so $K=1$. Then $s(100)=2+K e^{-100\ \ / 25}$. Then $3=s(0)=2+K e^{0}=2+K$, so $K=1$. Then $s(100)=2+K e^{-100 / 25}=2+1\
\ / 25}=2+1 \\cdot e^{-4}=2+e^{-4}$. The answer is (D)." \ \\cdot e^{-4}=2+e^{-4}$. The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_college_mathematics "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml
View file @ 2184b8de
dataset_name: college_medicine "dataset_name": "college_medicine"
description: "The following are multiple choice questions (with answers) about college\ "description": "The following are multiple choice questions (with answers) about college\
\ medicine.\n\nQ: An expected side effect of creatine supplementation is:\n(A) muscle\ \ medicine.\n\nQ: An expected side effect of creatine supplementation is:\n(A) muscle\
\ weakness. (B) gain in body mass. (C) muscle cramps. (D) loss of electrolytes.\n\ \ weakness. (B) gain in body mass. (C) muscle cramps. (D) loss of electrolytes.\n\
A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\ A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
...@@ -9,44 +9,44 @@ description: "The following are multiple choice questions (with answers) about c ...@@ -9,44 +9,44 @@ description: "The following are multiple choice questions (with answers) about c
\ endurance runners have a high proportion of Type I fibres in their leg muscles\ \ endurance runners have a high proportion of Type I fibres in their leg muscles\
\ (C) Liver glycogen is important in the maintenance of the blood glucose concentration\ \ (C) Liver glycogen is important in the maintenance of the blood glucose concentration\
\ (D) Insulin promotes glucose uptake by all tissues in the body\nA: Let's think\ \ (D) Insulin promotes glucose uptake by all tissues in the body\nA: Let's think\
\ step by step. We refer to Wikipedia articles on medicine for help. Let\u2019s\ \ step by step. We refer to Wikipedia articles on medicine for help. Let’s solve\
\ solve this step by step and go over each choice: \n(A) \u201CMuscle glycogen is\ \ this step by step and go over each choice: \n(A) “Muscle glycogen is broken down\
\ broken down enzymatically to glucose-1-phosphate\u201D: This is a correct statement.\n\ \ enzymatically to glucose-1-phosphate”: This is a correct statement.\n(B) “Elite\
(B) \u201CElite endurance runners have a high proportion of Type I fibres in their\ \ endurance runners have a high proportion of Type I fibres in their leg muscles”:\
\ leg muscles\u201D: This is a correct statement.\n(C) \u201CLiver glycogen is important\ \ This is a correct statement.\n(C) “Liver glycogen is important in the maintenance\
\ in the maintenance of the blood glucose concentration\u201D: This is a correct\ \ of the blood glucose concentration”: This is a correct statement. \n(D) “Insulin\
\ statement. \n(D) \u201CInsulin promotes glucose uptake by all tissues in the body\u201D\ \ promotes glucose uptake by all tissues in the body”: This is not a correct statement,\
: This is not a correct statement, because insulin promotes glucose uptake by the\ \ because insulin promotes glucose uptake by the liver, adipose tissue, and muscle,\
\ liver, adipose tissue, and muscle, but not all tissues. For instance, the tissues\ \ but not all tissues. For instance, the tissues in the brain and red blood cells\
\ in the brain and red blood cells are not affected by insulin. The answer is (D).\n\ \ are not affected by insulin. The answer is (D).\n\nQ: A high school science teacher\
\nQ: A high school science teacher fills a 1 liter bottle with pure nitrogen and\ \ fills a 1 liter bottle with pure nitrogen and seals the lid. The pressure is 1.70\
\ seals the lid. The pressure is 1.70 atm, and the room temperature is 25\xB0C.\ \ atm, and the room temperature is 25°C. Which two variables will both increase\
\ Which two variables will both increase the pressure of the system, if all other\ \ the pressure of the system, if all other variables are held constant?\n(A) Increasing\
\ variables are held constant?\n(A) Increasing temperature, increasing moles of\ \ temperature, increasing moles of gas (B) Increasing temperature, increasing volume\
\ gas (B) Increasing temperature, increasing volume (C) Decreasing volume, decreasing\ \ (C) Decreasing volume, decreasing temperature (D) Decreasing moles of gas, increasing\
\ temperature (D) Decreasing moles of gas, increasing volume\nA: Let's think step\ \ volume\nA: Let's think step by step. We refer to Wikipedia articles on medicine\
\ by step. We refer to Wikipedia articles on medicine for help. The relevant equation\ \ for help. The relevant equation for this is the ideal gas law: PV=nRT. To increase\
\ for this is the ideal gas law: PV=nRT. To increase the pressure of the system\ \ the pressure of the system (P), then either n (number of moles of the gas) or\
\ (P), then either n (number of moles of the gas) or T (temperature) have to increase.\ \ T (temperature) have to increase. The answer is (A).\n\nQ: In a genetic test of\
\ The answer is (A).\n\nQ: In a genetic test of a newborn, a rare genetic disorder\ \ a newborn, a rare genetic disorder is found that has X-linked recessive transmission.\
\ is found that has X-linked recessive transmission. Which of the following statements\ \ Which of the following statements is likely true regarding the pedigree of this\
\ is likely true regarding the pedigree of this disorder?\n(A) All descendants on\ \ disorder?\n(A) All descendants on the maternal side will have the disorder. (B)\
\ the maternal side will have the disorder. (B) Females will be approximately twice\ \ Females will be approximately twice as affected as males in this family. (C) All\
\ as affected as males in this family. (C) All daughters of an affected male will\ \ daughters of an affected male will be affected. (D) There will be equal distribution\
\ be affected. (D) There will be equal distribution of males and females affected.\n\ \ of males and females affected.\nA: Let's think step by step. We refer to Wikipedia\
A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\ \ articles on medicine for help. Let’s solve this step by step. Let's recall first\
\ Let\u2019s solve this step by step. Let's recall first that females have two X\ \ that females have two X chromosomes, while males have one X and one Y chromosome.\
\ chromosomes, while males have one X and one Y chromosome. This is an important\ \ This is an important fact we need to know before answering this question. \nBecause\
\ fact we need to know before answering this question. \nBecause a male can only\ \ a male can only pass his only one X chromosome to a daughter, if he is affected\
\ pass his only one X chromosome to a daughter, if he is affected by this rare genetic\ \ by this rare genetic disorder, then we know for sure that he will pass this rare\
\ disorder, then we know for sure that he will pass this rare genetic disorder to\ \ genetic disorder to all his future-born daughters. Therefore, “(C): All daughters\
\ all his future-born daughters. Therefore, \u201C(C): All daughters of an affected\ \ of an affected male will be affected” is a correct statement. The answer is (C).\n\
\ male will be affected\u201D is a correct statement. The answer is (C).\n\nQ: Glucose\ \nQ: Glucose is transported into the muscle cell:\n(A) via protein transporters\
\ is transported into the muscle cell:\n(A) via protein transporters called GLUT4.\ \ called GLUT4. (B) only in the presence of insulin. (C) via hexokinase. (D) via\
\ (B) only in the presence of insulin. (C) via hexokinase. (D) via monocarbylic\ \ monocarbylic acid transporters.\nA: Let's think step by step. We refer to Wikipedia\
\ acid transporters.\nA: Let's think step by step. We refer to Wikipedia articles\ \ articles on medicine for help. Glucose (also known as the blood sugar) is the\
\ on medicine for help. Glucose (also known as the blood sugar) is the main sugar\ \ main sugar found in the human body. It is transported into the muscle cell via\
\ found in the human body. It is transported into the muscle cell via diffusion\ \ diffusion through protein transporters called GLUT4. The answer is (A)."
\ through protein transporters called GLUT4. The answer is (A)." "group": "mmlu_flan_cot_fewshot_other"
include: _mmlu_flan_cot_fewshot_template_yaml "include": "_mmlu_flan_cot_fewshot_template_yaml"
task: mmlu_flan_cot_fewshot_college_medicine "task": "mmlu_flan_cot_fewshot_college_medicine"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml
View file @ 2184b8de
dataset_name: college_physics "dataset_name": "college_physics"
description: 'The following are multiple choice questions (with answers) about college "description": "The following are multiple choice questions (with answers) about college\
physics. \ physics.\n\nQ: A refracting telescope consists of two converging lenses separated\
\ by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification\
\ of the telescope is\n(A) 4 (B) 5 (C) 6 (D) 20\nA: Let's think step by step. In\
Q: A refracting telescope consists of two converging lenses separated by 100 cm. \ a refracting telescope, if both lenses are converging, the focus of both lenses\
The eye-piece lens has a focal length of 20 cm. The angular magnification of the \ must be between the two lenses, and thus the focal lengths of the two lenses must\
telescope is \ add up to their separation. Since the focal length of one lens is 20 cm, the focal\
\ length of the other must be 80 cm. The magnification is the ratio of these two\
(A) 4 (B) 5 (C) 6 (D) 20 \ focal lengths, or 4. The answer is (A).\n\nQ: The muon decays with a characteristic\
\ lifetime of about 10^-6 second into an electron, a muon neutrino, and an electron\
A: Let''s think step by step. In a refracting telescope, if both lenses are converging, \ antineutrino. The muon is forbidden from decaying into an electron and just a\
the focus of both lenses must be between the two lenses, and thus the focal lengths \ single neutrino by the law of conservation of\n(A) charge (B) mass (C) energy\
of the two lenses must add up to their separation. Since the focal length of one \ and momentum (D) lepton number\nA: Let's think step by step. Lepton number must\
lens is 20 cm, the focal length of the other must be 80 cm. The magnification is \ be conserved, meaning the total number of leptons minus the number of antileptons.\
the ratio of these two focal lengths, or 4. The answer is (A). \ If a muon decays into an electron and a single neutrino, the total lepton number\
\ would go from one to two, violating lepton number conservation. The answer is\
\ (D).\n\nQ: One end of a Nichrome wire of length 2L and cross-sectional area A\
Q: The muon decays with a characteristic lifetime of about 10^-6 second into an \ is attached to an end of another Nichrome wire of length L and cross- sectional\
electron, a muon neutrino, and an electron antineutrino. The muon is forbidden from \ area 2A. If the free end of the longer wire is at an electric potential of 8.0\
decaying into an electron and just a single neutrino by the law of conservation \ volts, and the free end of the shorter wire is at an electric potential of 1.0\
of \ volt, the potential at the junction of the two wires is most nearly equal to\n\
(A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V\nA: Let's think step by step. This is a\
(A) charge (B) mass (C) energy and momentum (D) lepton number \ simple voltage divider problem, where the longer wire has a resistance four times\
\ that of the shorter end. So the voltage divider ratio is 1 / 5, meaning that the\
A: Let''s think step by step. Lepton number must be conserved, meaning the total \ potential in the middle is 1.0 V + (8.0 V - 1.0 V) * 1/5 = 2.4 V. The answer is\
number of leptons minus the number of antileptons. If a muon decays into an electron \ (A).\n\nQ: A refracting telescope consists of two converging lenses separated\
and a single neutrino, the total lepton number would go from one to two, violating \ by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification\
lepton number conservation. The answer is (D). \ of the telescope is\n(A) 4 (B) 5 (C) 6 (D) 20\nA: Let's think step by step. In\
\ a refracting telescope, if both lenses are converging, the focus of both lenses\
\ must be between the two lenses, and thus the focal lengths of the two lenses must\
Q: One end of a Nichrome wire of length 2L and cross-sectional area A is attached \ add up to their separation. Since the focal length of one lens is 20 cm, the focal\
to an end of another Nichrome wire of length L and cross- sectional area 2A. If \ length of the other must be 80 cm. The magnification is the ratio of these two\
the free end of the longer wire is at an electric potential of 8.0 volts, and the \ focal lengths, or 4. The answer is (A).\n\nQ: For which of the following thermodynamic\
free end of the shorter wire is at an electric potential of 1.0 volt, the potential \ processes is the increase in the internal energy of an ideal gas equal to the\
at the junction of the two wires is most nearly equal to \ heat added to the gas?\n(A) Constant temperature (B) Constant volume (C) Constant\
\ pressure (D) Adiabatic\nA: Let's think step by step. Heat added to the gas can\
(A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V \ go into the gases internal energy or work done against an external force. However,\
\ if the volume of the gas container is constant, no work will be done (since work\
A: Let''s think step by step. This is a simple voltage divider problem, where the \ is pressure times change in volume). So, at constant volume, all of the heat goes\
longer wire has a resistance four times that of the shorter end. So the voltage \ into the internal energy. The answer is (B)."
divider ratio is 1 / 5, meaning that the potential in the middle is 1.0 V + (8.0 "group": "mmlu_flan_cot_fewshot_stem"
V - 1.0 V) * 1/5 = 2.4 V. The answer is (A). "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_college_physics"
Q: A refracting telescope consists of two converging lenses separated by 100 cm.
The eye-piece lens has a focal length of 20 cm. The angular magnification of the
telescope is
(A) 4 (B) 5 (C) 6 (D) 20
A: Let''s think step by step. In a refracting telescope, if both lenses are converging,
the focus of both lenses must be between the two lenses, and thus the focal lengths
of the two lenses must add up to their separation. Since the focal length of one
lens is 20 cm, the focal length of the other must be 80 cm. The magnification is
the ratio of these two focal lengths, or 4. The answer is (A).
Q: For which of the following thermodynamic processes is the increase in the internal
energy of an ideal gas equal to the heat added to the gas?
(A) Constant temperature (B) Constant volume (C) Constant pressure (D) Adiabatic
A: Let''s think step by step. Heat added to the gas can go into the gases internal
energy or work done against an external force. However, if the volume of the gas
container is constant, no work will be done (since work is pressure times change
in volume). So, at constant volume, all of the heat goes into the internal energy.
The answer is (B).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_college_physics
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml
View file @ 2184b8de
dataset_name: computer_security "dataset_name": "computer_security"
description: "The following are multiple choice questions (with answers) about computer\ "description": "The following are multiple choice questions (with answers) about computer\
\ security.\n\nQ: SHA-1 has a message digest of\n(A) 160 bits (B) 512 bits (C) 628\ \ security.\n\nQ: SHA-1 has a message digest of\n(A) 160 bits (B) 512 bits (C) 628\
\ bits (D) 820 bits\nA: Let's think step by step. Since SHA-1 is a hash function\ \ bits (D) 820 bits\nA: Let's think step by step. Since SHA-1 is a hash function\
\ which takes an input and produces a 160-bit (20-byte) hash value, its message\ \ which takes an input and produces a 160-bit (20-byte) hash value, its message\
\ digest is 160 bits. The answer is (A).\n\nQ: _____________ can modify data on\ \ digest is 160 bits. The answer is (A).\n\nQ: _____________ can modify data on\
\ your system \u2013 so that your system doesn\u2019t run correctly or you can no\ \ your system – so that your system doesn’t run correctly or you can no longer access\
\ longer access specific data, or it may even ask for ransom in order to give your\ \ specific data, or it may even ask for ransom in order to give your access.\n(A)\
\ access.\n(A) IM \u2013 Trojans (B) Backdoor Trojans (C) Trojan-Downloader (D)\ \ IM – Trojans (B) Backdoor Trojans (C) Trojan-Downloader (D) Ransom Trojan\nA:\
\ Ransom Trojan\nA: Let's think step by step. The system is asking for trojans,\ \ Let's think step by step. The system is asking for trojans, which are for ransom,\
\ which are for ransom, which means ransom trojan. The answer is (D).\n\nQ: What\ \ which means ransom trojan. The answer is (D).\n\nQ: What is ethical hacking?\n\
\ is ethical hacking?\n(A) \"Hacking\" ethics so they justify unintended selfish\ (A) \"Hacking\" ethics so they justify unintended selfish behavior (B) Hacking systems\
\ behavior (B) Hacking systems (e.g., during penetration testing) to expose vulnerabilities\ \ (e.g., during penetration testing) to expose vulnerabilities so they can be fixed,\
\ so they can be fixed, rather than exploited (C) Hacking into systems run by those\ \ rather than exploited (C) Hacking into systems run by those whose ethics you disagree\
\ whose ethics you disagree with (D) A slang term for rapid software development,\ \ with (D) A slang term for rapid software development, e.g., as part of hackathons\n\
\ e.g., as part of hackathons\nA: Let's think step by step. Ethical hacking is a\ A: Let's think step by step. Ethical hacking is a process of detecting vulnerabilities\
\ process of detecting vulnerabilities in an application, system, or organization's\ \ in an application, system, or organization's infrastructure that an attacker can\
\ infrastructure that an attacker can use to exploit an individual or organization.\ \ use to exploit an individual or organization. They use this process to prevent\
\ They use this process to prevent cyberattacks and security breaches by lawfully\ \ cyberattacks and security breaches by lawfully hacking into the systems and looking\
\ hacking into the systems and looking for weak points. The answer is (B).\n\nQ:\ \ for weak points. The answer is (B).\n\nQ: The ____________ is anything which your\
\ The ____________ is anything which your search engine cannot search.\n(A) Haunted\ \ search engine cannot search.\n(A) Haunted web (B) World Wide Web (C) Surface web\
\ web (B) World Wide Web (C) Surface web (D) Deep Web\nA: Let's think step by step.\ \ (D) Deep Web\nA: Let's think step by step. The search engine searches on the Surface\
\ The search engine searches on the Surface Web, which is the portion of the world\ \ Web, which is the portion of the world wide web which is visible so (B,C) are\
\ wide web which is visible so (B,C) are wrong. The Haunted Web doesn\u2019t correspond\ \ wrong. The Haunted Web doesn’t correspond to an internet concept. The Deep Web\
\ to an internet concept. The Deep Web is the part of the World Wide Web which is\ \ is the part of the World Wide Web which is not indexed. The answer is (D).\n\n\
\ not indexed. The answer is (D).\n\nQ: Exploitation of the Heartbleed bug permits\n\ Q: Exploitation of the Heartbleed bug permits\n(A) overwriting cryptographic keys\
(A) overwriting cryptographic keys in memory (B) a kind of code injection (C) a\ \ in memory (B) a kind of code injection (C) a read outside bounds of a buffer (D)\
\ read outside bounds of a buffer (D) a format string attack\nA: Let's think step\ \ a format string attack\nA: Let's think step by step. The Heartbleed Bug is a serious\
\ by step. The Heartbleed Bug is a serious vulnerability in the popular OpenSSL\ \ vulnerability in the popular OpenSSL cryptographic software library. Heartbleed\
\ cryptographic software library. Heartbleed resulted from improper input validation\ \ resulted from improper input validation (due to a missing bounds check) in the\
\ (due to a missing bounds check) in the implementation of the TLS heartbeat extension.\ \ implementation of the TLS heartbeat extension. The vulnerability was classified\
\ The vulnerability was classified as a buffer over-read, a situation where more\ \ as a buffer over-read, a situation where more data can be read than should be\
\ data can be read than should be allowed. The answer is (C)." \ allowed. The answer is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_computer_security "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_computer_security"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml
View file @ 2184b8de
dataset_name: conceptual_physics "dataset_name": "conceptual_physics"
description: "\nThe following are multiple choice questions (with answers) about conceptual\ "description": "\nThe following are multiple choice questions (with answers) about\
\ physics.\n\nQ: Colors in a soap bubble result from light\n(A) converted to a different\ \ conceptual physics.\n\nQ: Colors in a soap bubble result from light\n(A) converted\
\ frequency (B) deflection (C) interference (D) polarization\nA: Let's think step\ \ to a different frequency (B) deflection (C) interference (D) polarization\nA:\
\ by step. In a soap bubble film, the light bounces between the two soap-air interfaces\ \ Let's think step by step. In a soap bubble film, the light bounces between the\
\ many times, interfering with itself constructively or destructively depending\ \ two soap-air interfaces many times, interfering with itself constructively or\
\ on the width of the film. This results in different colors being visible. The\ \ destructively depending on the width of the film. This results in different colors\
\ answer is (C).\n\nQ: Compared with the mass of a uranium atom undergoing fission,\ \ being visible. The answer is (C).\n\nQ: Compared with the mass of a uranium atom\
\ the combined masses of the products after fission are\n(A) less (B) more (C) the\ \ undergoing fission, the combined masses of the products after fission are\n(A)\
\ same (D) zero\nA: Let's think step by step. Fission releases energy, which comes\ \ less (B) more (C) the same (D) zero\nA: Let's think step by step. Fission releases\
\ from the rest mass of its initial nucleus. Thus the mass of the products is less\ \ energy, which comes from the rest mass of its initial nucleus. Thus the mass of\
\ than the mass of the reactant uranium nucleus. The answer is (A).\n\nQ: Things\ \ the products is less than the mass of the reactant uranium nucleus. The answer\
\ that are equivalent according to the equivalence principle are\n(A) space and\ \ is (A).\n\nQ: Things that are equivalent according to the equivalence principle\
\ time. (B) a traveling twin and a stay-at-home twin. (C) gravity and acceleration.\ \ are\n(A) space and time. (B) a traveling twin and a stay-at-home twin. (C) gravity\
\ (D) mass and energy.\nA: Let's think step by step. Einstein\u2019s famous equivalence\ \ and acceleration. (D) mass and energy.\nA: Let's think step by step. Einstein’s\
\ principle states that gravity and acceleration are equivalent. The answer is (C).\n\ \ famous equivalence principle states that gravity and acceleration are equivalent.\
\nQ: Which of these three elements has the most mass per nucleon?\n(A) Hydrogen\ \ The answer is (C).\n\nQ: Which of these three elements has the most mass per nucleon?\n\
\ (B) Iron (C) Uranium (D) Same in each\nA: Let's think step by step. Due to nuclear\ (A) Hydrogen (B) Iron (C) Uranium (D) Same in each\nA: Let's think step by step.\
\ binding energy, the mass of an atomic nucleus is less than the sum of individual\ \ Due to nuclear binding energy, the mass of an atomic nucleus is less than the\
\ masses of the free constituent protons and neutrons; this is known as the mass\ \ sum of individual masses of the free constituent protons and neutrons; this is\
\ defect. Hydrogen has no mass defect because it has only a single nucleon, so it\ \ known as the mass defect. Hydrogen has no mass defect because it has only a single\
\ will have the most mass per nucleon. The answer is (A).\n\nQ: A model airplane\ \ nucleon, so it will have the most mass per nucleon. The answer is (A).\n\nQ: A\
\ flies slower when flying into the wind and faster with wind at its back. When\ \ model airplane flies slower when flying into the wind and faster with wind at\
\ launched at right angles to the wind a cross wind its groundspeed compared with\ \ its back. When launched at right angles to the wind a cross wind its groundspeed\
\ flying in still air is\n(A) the same (B) greater (C) less (D) either greater or\ \ compared with flying in still air is\n(A) the same (B) greater (C) less (D) either\
\ less depending on wind speed\nA: Let's think step by step. The plane\u2019s speed\ \ greater or less depending on wind speed\nA: Let's think step by step. The plane’s\
\ in the direction of the wind is greater than it would be in the absence of wind,\ \ speed in the direction of the wind is greater than it would be in the absence\
\ and its direction orthogonal to the wind is the same as it would be in the absence\ \ of wind, and its direction orthogonal to the wind is the same as it would be in\
\ of the wind. The total speed, which is these two components added in quadrature,\ \ the absence of the wind. The total speed, which is these two components added\
\ is thus greater than the speed in still air. The answer is (B)." \ in quadrature, is thus greater than the speed in still air. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_conceptual_physics "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_conceptual_physics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml
View file @ 2184b8de
dataset_name: econometrics "dataset_name": "econometrics"
description: "The following are multiple choice questions (with answers) about econometrics.\n\ "description": "The following are multiple choice questions (with answers) about econometrics.\n\
\nQ: Suppose now that a researcher wishes to use information criteria to determine\ \nQ: Suppose now that a researcher wishes to use information criteria to determine\
\ the optimal lag length for a VAR. 500 observations are available for the bi-variate\ \ the optimal lag length for a VAR. 500 observations are available for the bi-variate\
\ VAR, and the values of the determinant of the variance-covariance matrix of residuals\ \ VAR, and the values of the determinant of the variance-covariance matrix of residuals\
\ are 0.0336, 0.0169, 0.0084, and 0.0062 for 1, 2, 3, and 4 lags respectively. What\ \ are 0.0336, 0.0169, 0.0084, and 0.0062 for 1, 2, 3, and 4 lags respectively. What\
\ is the optimal model order according to Akaike's information criterion?\n(A) 1\ \ is the optimal model order according to Akaike's information criterion?\n(A) 1\
\ lag (B) 2 lags (C) 3 lags (D) 4 lags\nA: Let's think step by step. We refer to\ \ lag (B) 2 lags (C) 3 lags (D) 4 lags\nA: Let's think step by step. We refer to\
\ Wikipedia articles on econometrics for help. Let\u2019s solve this problem step\ \ Wikipedia articles on econometrics for help. Let’s solve this problem step by\
\ by step. First of all, let\u2019s recall that for a given set of data, Akaike's\ \ step. First of all, let’s recall that for a given set of data, Akaike's information\
\ information criterion (AIC) allows us to measure how well a statistical model\ \ criterion (AIC) allows us to measure how well a statistical model fits the data;\
\ fits the data; it is an estimator of prediction error. Here in this problem we\ \ it is an estimator of prediction error. Here in this problem we will need to use\
\ will need to use the formula ln(det(sigma_hat)) + (2 * k / T) to determine the\ \ the formula ln(det(sigma_hat)) + (2 * k / T) to determine the values of Akaike’s\
\ values of Akaike\u2019s criterion, where ln denotes the natural log function,\ \ criterion, where ln denotes the natural log function, det the determinant function,\
\ det the determinant function, k the total number of parameters in total (across\ \ k the total number of parameters in total (across both equations), and T the number\
\ both equations), and T the number of observations (which, in this case, is equal\ \ of observations (which, in this case, is equal to 500). For 1 lag, the number\
\ to 500). For 1 lag, the number of parameters in total is equal to 6; for 2 lags,\ \ of parameters in total is equal to 6; for 2 lags, it is 10; for 3 lags, it is\
\ it is 10; for 3 lags, it is 14; and for 4 lags, it is 18. Now, let\u2019s calculate\ \ 14; and for 4 lags, it is 18. Now, let’s calculate the values of the criterion\
\ the values of the criterion for each lag:\n(A) 1 lag: ln(0.0336) + (2 * 6 / 500)\ \ for each lag:\n(A) 1 lag: ln(0.0336) + (2 * 6 / 500) = ln(0.0336) + (12 / 500)\
\ = ln(0.0336) + (12 / 500) = -3.369\n(B) 2 lags: ln(0.0169) + (2 * 10 / 500) =\ \ = -3.369\n(B) 2 lags: ln(0.0169) + (2 * 10 / 500) = ln(0.0169) + (20 / 500) =\
\ ln(0.0169) + (20 / 500) = -4.040\n(C) 3 lags: ln(0.0084) + (2 * 14 / 500) = ln(0.0084)\ \ -4.040\n(C) 3 lags: ln(0.0084) + (2 * 14 / 500) = ln(0.0084) + (28 / 500) =-4.724\n\
\ + (28 / 500) =-4.724\n(D) 4 lags: ln(0.0062) + (2 * 18 / 500) = ln(0.0062) + (36\ (D) 4 lags: ln(0.0062) + (2 * 18 / 500) = ln(0.0062) + (36 / 500) =-5.011\nBecause\
\ / 500) =-5.011\nBecause the optimal model order according to AIC minimizes the\ \ the optimal model order according to AIC minimizes the information criterion,\
\ information criterion, the answer should be the one with the lowest value. In\ \ the answer should be the one with the lowest value. In this case, (D) has the\
\ this case, (D) has the lowest value. The answer is (C).\n\nQ: Consider the following\ \ lowest value. The answer is (C).\n\nQ: Consider the following AR(1) model with\
\ AR(1) model with the disturbances having zero mean and unit variance\nyt = 0.2\ \ the disturbances having zero mean and unit variance\nyt = 0.2 + 0.4 yt-1 + ut\n\
\ + 0.4 yt-1 + ut\nThe (unconditional) mean of y will be given by\n(A) 0.2 (B) 0.4\ The (unconditional) mean of y will be given by\n(A) 0.2 (B) 0.4 (C) 0.5 (D) 0.33\n\
\ (C) 0.5 (D) 0.33\nA: Let's think step by step. We refer to Wikipedia articles\ A: Let's think step by step. We refer to Wikipedia articles on econometrics for\
\ on econometrics for help. Let\u2019s solve this problem step by step. If we have\ \ help. Let’s solve this problem step by step. If we have a an AR(1) model with\
\ a an AR(1) model with the disturbances having zero mean and unit variance, then\ \ the disturbances having zero mean and unit variance, then the unconditional mean\
\ the unconditional mean of y is equal to the following:\nunconditional mean of\ \ of y is equal to the following:\nunconditional mean of y = (the intercept term)\
\ y = (the intercept term) / (1 - autoregressive coefficient)\nWe know that the\ \ / (1 - autoregressive coefficient)\nWe know that the intercept term is 0.2 and\
\ intercept term is 0.2 and the autoregressive coefficient is 0.4; thus, we have:\n\ \ the autoregressive coefficient is 0.4; thus, we have:\nunconditional mean of y\
unconditional mean of y = (0.2) / (1 - 0.4) = (0.2) / (0.6) = 2 / 6 = 1 / 3, which\ \ = (0.2) / (1 - 0.4) = (0.2) / (0.6) = 2 / 6 = 1 / 3, which is approximately 0.33.\
\ is approximately 0.33. That means that the answer should be (D) 0.33. The answer\ \ That means that the answer should be (D) 0.33. The answer is (D).\n\nQ: What would\
\ is (D).\n\nQ: What would be then consequences for the OLS estimator if heteroscedasticity\ \ be then consequences for the OLS estimator if heteroscedasticity is present in\
\ is present in a regression model but ignored?\n(A) It will be biased (B) It will\ \ a regression model but ignored?\n(A) It will be biased (B) It will be inconsistent\
\ be inconsistent (C) It will be inefficient (D) All of (a), (b) and (c) will be\ \ (C) It will be inefficient (D) All of (a), (b) and (c) will be true.\nA: Let's\
\ true.\nA: Let's think step by step. We refer to Wikipedia articles on econometrics\ \ think step by step. We refer to Wikipedia articles on econometrics for help. Heteroscedasticity\
\ for help. Heteroscedasticity refers to the condition where the variance of the\ \ refers to the condition where the variance of the error terms is not constant\
\ error terms is not constant across multiple observations. If heteroscedasticity\ \ across multiple observations. If heteroscedasticity is present in a regression\
\ is present in a regression model, then the coefficient estimates in the OLS estimator\ \ model, then the coefficient estimates in the OLS estimator will be not only unbiased\
\ will be not only unbiased and consistent but also inefficient. Because (A) and\ \ and consistent but also inefficient. Because (A) and (B) are incorrect choices\
\ (B) are incorrect choices and (C) is a correct choice, (D) cannot be the right\ \ and (C) is a correct choice, (D) cannot be the right answer. Ultimately, (C) is\
\ answer. Ultimately, (C) is the only true choice. The answer is (C).\n\nQ: Suppose\ \ the only true choice. The answer is (C).\n\nQ: Suppose that a test statistic has\
\ that a test statistic has associated with it a p-value of 0.08. Which one of the\ \ associated with it a p-value of 0.08. Which one of the following statements is\
\ following statements is true?\n(i) If the size of the test were exactly 8%, we\ \ true?\n(i) If the size of the test were exactly 8%, we would be indifferent between\
\ would be indifferent between rejecting and not rejecting the null hypothesis\n\ \ rejecting and not rejecting the null hypothesis\n(ii) The null would be rejected\
(ii) The null would be rejected if a 10% size of test were used\n(iii) The null\ \ if a 10% size of test were used\n(iii) The null would not be rejected if a 1%\
\ would not be rejected if a 1% size of test were used\n(iv) The null would be rejected\ \ size of test were used\n(iv) The null would be rejected if a 5% size of test were\
\ if a 5% size of test were used.\n(A) (ii) and (iv) only (B) (i) and (iii) only\ \ used.\n(A) (ii) and (iv) only (B) (i) and (iii) only (C) (i), (ii), and (iii)\
\ (C) (i), (ii), and (iii) only (D) (i), (ii), (iii), and (iv).\nA: Let's think\ \ only (D) (i), (ii), (iii), and (iv).\nA: Let's think step by step. We refer to\
\ step by step. We refer to Wikipedia articles on econometrics for help. Let\u2019\ \ Wikipedia articles on econometrics for help. Let’s reason about each of the options.\n\
s reason about each of the options.\n(i) is a true statement.\n(ii) is a true statement.\n\ (i) is a true statement.\n(ii) is a true statement.\n(iii) is a true statement.\n\
(iii) is a true statement.\n(iv) is not a true statement. Thus, (i), (ii), and (iii)\ (iv) is not a true statement. Thus, (i), (ii), and (iii) are true. The answer is\
\ are true. The answer is (C).\n\nQ: For a stationary autoregressive process, shocks\ \ (C).\n\nQ: For a stationary autoregressive process, shocks will\n(A) Eventually\
\ will\n(A) Eventually die away (B) Persist indefinitely (C) Grow exponentially\ \ die away (B) Persist indefinitely (C) Grow exponentially (D) Never occur\nA: Let's\
\ (D) Never occur\nA: Let's think step by step. We refer to Wikipedia articles on\ \ think step by step. We refer to Wikipedia articles on econometrics for help. This\
\ econometrics for help. This is a formal logic problem about stationally process.\ \ is a formal logic problem about stationally process. For a stationary autoregressive\
\ For a stationary autoregressive process, shocks will eventually die away. The\ \ process, shocks will eventually die away. The answer is (A)."
\ answer is (A)." "group": "mmlu_flan_cot_fewshot_social_sciences"
include: _mmlu_flan_cot_fewshot_template_yaml "include": "_mmlu_flan_cot_fewshot_template_yaml"
task: mmlu_flan_cot_fewshot_econometrics "task": "mmlu_flan_cot_fewshot_econometrics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml
View file @ 2184b8de
dataset_name: electrical_engineering "dataset_name": "electrical_engineering"
description: "\nThe following are multiple choice questions (with answers) about electrical\ "description": "\nThe following are multiple choice questions (with answers) about\
\ engineering.\n\nQ: A point pole has a strength of 4\u03C0 * 10^-4 weber. The force\ \ electrical engineering.\n\nQ: A point pole has a strength of 4π * 10^-4 weber.\
\ in newtons on a point pole of 4\u03C0 * 1.5 * 10^-4 weber placed at a distance\ \ The force in newtons on a point pole of 4π * 1.5 * 10^-4 weber placed at a distance\
\ of 10 cm from it will be\n(A) 15 N. (B) 20 N. (C) 7.5 N. (D) 3.75 N.\nA: Let's\ \ of 10 cm from it will be\n(A) 15 N. (B) 20 N. (C) 7.5 N. (D) 3.75 N.\nA: Let's\
\ think step by step. The force between two point poles is given by m_1m_2/(mu_0\ \ think step by step. The force between two point poles is given by m_1m_2/(mu_0\
\ 4 \\pi r^2), in analogy to Coulomb\u2019s law. Plugging in the values given in\ \ 4 \\pi r^2), in analogy to Coulomb’s law. Plugging in the values given in the\
\ the question, we calculate that the force is approximately 15 N. The answer is\ \ question, we calculate that the force is approximately 15 N. The answer is (A).\n\
\ (A).\n\nQ: The coil of a moving coil meter has 100 turns, is 40 mm long and 30\ \nQ: The coil of a moving coil meter has 100 turns, is 40 mm long and 30 mm wide.\
\ mm wide. The control torque is 240*10-6 N-m on full scale. If magnetic flux density\ \ The control torque is 240*10-6 N-m on full scale. If magnetic flux density is\
\ is 1Wb/m2 range of meter is\n(A) 1 mA. (B) 2 mA. (C) 3 mA. (D) 4 mA.\nA: Let's\ \ 1Wb/m2 range of meter is\n(A) 1 mA. (B) 2 mA. (C) 3 mA. (D) 4 mA.\nA: Let's think\
\ think step by step. The torque on a coil in a uniform magnetic field is given\ \ step by step. The torque on a coil in a uniform magnetic field is given by BANI,\
\ by BANI, where B is the magnetic flux density, A is the area of the coil, N is\ \ where B is the magnetic flux density, A is the area of the coil, N is the number\
\ the number of turns, and I is the current. So we have that I = (Torque)/(BAN),\ \ of turns, and I is the current. So we have that I = (Torque)/(BAN), or 240e-6/(1200e-6\
\ or 240e-6/(1200e-6 * 100 * 1) = 2e-3. The answer is (B).\n\nQ: In an SR latch\ \ * 100 * 1) = 2e-3. The answer is (B).\n\nQ: In an SR latch built from NOR gates,\
\ built from NOR gates, which condition is not allowed\n(A) S=0, R=0 (B) S=0, R=1\ \ which condition is not allowed\n(A) S=0, R=0 (B) S=0, R=1 (C) S=1, R=0 (D) S=1,\
\ (C) S=1, R=0 (D) S=1, R=1\nA: Let's think step by step. An SR latch is a set-reset\ \ R=1\nA: Let's think step by step. An SR latch is a set-reset latch; in the case\
\ latch; in the case where S=1 and R=1, the circuit has no stable state; instead\ \ where S=1 and R=1, the circuit has no stable state; instead a race condition will\
\ a race condition will be produced within the circuit, so the device will be in\ \ be produced within the circuit, so the device will be in an undefined state. So\
\ an undefined state. So S=1, R=1 is an illegal input. The answer is (D).\n\nQ:\ \ S=1, R=1 is an illegal input. The answer is (D).\n\nQ: Two long parallel conductors\
\ Two long parallel conductors carry 100 A. If the conductors are separated by 20\ \ carry 100 A. If the conductors are separated by 20 mm, the force per meter of\
\ mm, the force per meter of length of each conductor will be\n(A) 100 N. (B) 0.1\ \ length of each conductor will be\n(A) 100 N. (B) 0.1 N. (C) 1 N. (D) 0.01 N.\n\
\ N. (C) 1 N. (D) 0.01 N.\nA: Let's think step by step. The magnetic force-per-length\ A: Let's think step by step. The magnetic force-per-length between two current-carrying\
\ between two current-carrying conductors is given by \\mu_0 I_1 I_2 / (2 \\pi r),\ \ conductors is given by \\mu_0 I_1 I_2 / (2 \\pi r), where $r$ is the separation\
\ where $r$ is the separation distance and I_1 and I_2 are the currents. Plugging\ \ distance and I_1 and I_2 are the currents. Plugging in 100 A for I_1 and I_2,\
\ in 100 A for I_1 and I_2, and 20 mm for r, gives 0.1 N. The answer is (B).\n\n\ \ and 20 mm for r, gives 0.1 N. The answer is (B).\n\nQ: In a 2 pole lap winding\
Q: In a 2 pole lap winding dc machine , the resistance of one conductor is 2\u03A9\ \ dc machine , the resistance of one conductor is 2Ω and total number of conductors\
\ and total number of conductors is 100. Find the total resistance\n(A) 200\u03A9\ \ is 100. Find the total resistance\n(A) 200Ω (B) 100Ω (C) 50Ω (D) 10Ω\nA: Let's\
\ (B) 100\u03A9 (C) 50\u03A9 (D) 10\u03A9\nA: Let's think step by step. In lap winding,\ \ think step by step. In lap winding, effectively two resistors are connected in\
\ effectively two resistors are connected in parallel, so the actual resistance\ \ parallel, so the actual resistance of each pair is 1 Ohm. Since we have 50 pairs,\
\ of each pair is 1 Ohm. Since we have 50 pairs, we get a total resistance of 50\ \ we get a total resistance of 50 Ohms. The answer is (C)."
\ Ohms. The answer is (C)." "group": "mmlu_flan_cot_fewshot_stem"
include: _mmlu_flan_cot_fewshot_template_yaml "include": "_mmlu_flan_cot_fewshot_template_yaml"
task: mmlu_flan_cot_fewshot_electrical_engineering "task": "mmlu_flan_cot_fewshot_electrical_engineering"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml
View file @ 2184b8de
dataset_name: elementary_mathematics "dataset_name": "elementary_mathematics"
description: "The following are multiple choice questions (with answers) about elementary\ "description": "The following are multiple choice questions (with answers) about elementary\
\ mathematics.\n\nQ: Olivia used the rule \"Add 11\" to create the number pattern\ \ mathematics.\n\nQ: Olivia used the rule \"Add 11\" to create the number pattern\
\ shown below. 10, 21, 32, 43, 54. Which statement about the number pattern is true?\n\ \ shown below. 10, 21, 32, 43, 54. Which statement about the number pattern is true?\n\
(A) The 10th number in the pattern will be an even number.\n(B) The number pattern\ (A) The 10th number in the pattern will be an even number.\n(B) The number pattern\
...@@ -22,19 +22,20 @@ description: "The following are multiple choice questions (with answers) about e ...@@ -22,19 +22,20 @@ description: "The following are multiple choice questions (with answers) about e
\ the other choices are incorrect. The answer is (A).\n\nQ: A store sells 107 different\ \ the other choices are incorrect. The answer is (A).\n\nQ: A store sells 107 different\
\ colors of paint. They have 25 cans of each color in storage. The number of cans\ \ colors of paint. They have 25 cans of each color in storage. The number of cans\
\ of paint the store has in storage can be found using the expression below. 107\ \ of paint the store has in storage can be found using the expression below. 107\
\ \xD7 25. How many cans of paint does the store have in storage?\n(A) 749\n(B)\ \ × 25. How many cans of paint does the store have in storage?\n(A) 749\n(B) 2,675\n\
\ 2,675\n(C) 2,945\n(D) 4,250\nA: Let's think step by step. We can calculate 107\ (C) 2,945\n(D) 4,250\nA: Let's think step by step. We can calculate 107 x 25 = (100\
\ x 25 = (100 x 25) + (7 x 25) = 2500 + 175 = 2675. The answer is (B).\n\nQ: A total\ \ x 25) + (7 x 25) = 2500 + 175 = 2675. The answer is (B).\n\nQ: A total of 30 players\
\ of 30 players will play basketball at a park. There will be exactly 5 players\ \ will play basketball at a park. There will be exactly 5 players on each team.\
\ on each team. Which statement correctly explains how to find the number of teams\ \ Which statement correctly explains how to find the number of teams needed?\n(A)\
\ needed?\n(A) Add 5 to 30 to find 35 teams.\n(B) Divide 30 by 5 to find 6 teams.\n\ \ Add 5 to 30 to find 35 teams.\n(B) Divide 30 by 5 to find 6 teams.\n(C) Multiply\
(C) Multiply 30 and 5 to find 150 teams.\n(D) Subtract 5 from 30 to find 25 teams.\n\ \ 30 and 5 to find 150 teams.\n(D) Subtract 5 from 30 to find 25 teams.\nA: Let's\
A: Let's think step by step. We want to find the number of teams. We know that there\ \ think step by step. We want to find the number of teams. We know that there are\
\ are 5 players/team, and 30 players. Thus to get the number of teams we divide\ \ 5 players/team, and 30 players. Thus to get the number of teams we divide players\
\ players by players/team, so 30 players / 5 players/team = 6 teams. The answer\ \ by players/team, so 30 players / 5 players/team = 6 teams. The answer is (B).\n\
\ is (B).\n\nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n\ \nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n(B) (5 x 5)\
(B) (5 x 5) + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think\ \ + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think step by\
\ step by step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5\ \ step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5 x 4). The\
\ x 4). The answer is (B)." \ answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_elementary_mathematics "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_elementary_mathematics"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml
View file @ 2184b8de
dataset_name: formal_logic "dataset_name": "formal_logic"
description: "The following are multiple choice questions (with answers) about formal\ "description": "The following are multiple choice questions (with answers) about formal\
\ logic.\n\nQ: Which of the given formulas of PL is the best symbolization of the\ \ logic.\n\nQ: Which of the given formulas of PL is the best symbolization of the\
\ following sentence?\nTurtles live long lives and are happy creatures, unless they\ \ following sentence?\nTurtles live long lives and are happy creatures, unless they\
\ are injured.\n(A) (L \u2022 H) \u2261 I (B) (L \u2022 H) \u2228 I (C) L \u2022\ \ are injured.\n(A) (L • H) ≡ I (B) (L • H) ∨ I (C) L • (H ∨ I) (D) L • (H ⊃ R).\n\
\ (H \u2228 I) (D) L \u2022 (H \u2283 R).\nA: Let's think step by step. We refer\ A: Let's think step by step. We refer to Wikipedia articles on formal logic for\
\ to Wikipedia articles on formal logic for help. Let\u2019s solve this step by\ \ help. Let’s solve this step by step. Let “L” denote “living long”, H “being happy”,\
\ step. Let \u201CL\u201D denote \u201Cliving long\u201D, H \u201Cbeing happy\u201D\ \ and “I” “being injured”. Now, consider each choice:\n(A) means (living long AND\
, and \u201CI\u201D \u201Cbeing injured\u201D. Now, consider each choice:\n(A) means\ \ being happy) is equivalent to (being injured). \n(B) means (living long AND being\
\ (living long AND being happy) is equivalent to (being injured). \n(B) means (living\ \ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\
\ long AND being happy) OR (being injured). \n(C) means (living long) AND (being\ \ injured). \n(D) means (living long) AND (being happy implies being R), but what\
\ happy OR being injured). \n(D) means (living long) AND (being happy implies being\ \ R denotes is not clear.\nObviously, (B) is the best symbolization of the original\
\ R), but what R denotes is not clear.\nObviously, (B) is the best symbolization\ \ sentence. The answer is (B).\n\nQ: Select the best translation into predicate\
\ of the original sentence. The answer is (B).\n\nQ: Select the best translation\ \ logic.George borrows Hector's lawnmower. (g: George; h: Hector; l: Hector's lawnmower;\
\ into predicate logic.George borrows Hector's lawnmower. (g: George; h: Hector;\ \ Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh (D) Bghl\nA: Let's think\
\ l: Hector's lawnmower; Bxyx: x borrows y from z).\n(A) Blgh (B) Bhlg (C) Bglh\ \ step by step. We refer to Wikipedia articles on formal logic for help. Let’s solve\
\ (D) Bghl\nA: Let's think step by step. We refer to Wikipedia articles on formal\ \ this step by step. We are told that “Bxyx” means “x borrows y from z”. We can\
\ logic for help. Let\u2019s solve this step by step. We are told that \u201CBxyx\u201D\ \ rewrite “George borrows Hector's lawnmower” as “George borrows a lawnmower from\
\ means \u201Cx borrows y from z\u201D. We can rewrite \u201CGeorge borrows Hector's\ \ Hector”, which can then be translated into predicate logic as “Bglh”. The answer\
\ lawnmower\u201D as \u201CGeorge borrows a lawnmower from Hector\u201D, which can\ \ “Bglh” appears in (C); therefore, (C) must be the correct answer. The answer is\
\ then be translated into predicate logic as \u201CBglh\u201D. The answer \u201C\ \ (C).\n\nQ: \nSelect the best English interpretation of the given arguments in\
Bglh\u201D appears in (C); therefore, (C) must be the correct answer. The answer\ \ predicate logic.\nDm\n(∀x)(Wx ⊃ ~Dx). \n(∀x)Wx ∨ Ag\t/ (∃x)Ax\n(A) Marina is a\
\ is (C).\n\nQ: \nSelect the best English interpretation of the given arguments\ \ dancer. Some weaklings are not dancers. Either everything is a weakling or Georgia\
\ in predicate logic.\nDm\n(\u2200x)(Wx \u2283 ~Dx). \n(\u2200x)Wx \u2228 Ag\t/\ \ plays volleyball. So something plays volleyball. (B) Marina is a dancer. No weakling\
\ (\u2203x)Ax\n(A) Marina is a dancer. Some weaklings are not dancers. Either everything\ \ is a dancer. Everything is either a weakling or plays volleyball. So something\
\ is a weakling or Georgia plays volleyball. So something plays volleyball. (B)\ \ plays volleyball. (C) Marina is a dancer. Some weaklings are not dancers. Everything\
\ Marina is a dancer. No weakling is a dancer. Everything is either a weakling or\ \ is either a weakling or plays volleyball. So something plays volleyball. (D) Marina\
\ plays volleyball. So something plays volleyball. (C) Marina is a dancer. Some\ \ is a dancer. No weakling is a dancer. Either everything is a weakling or Georgia\
\ weaklings are not dancers. Everything is either a weakling or plays volleyball.\ \ plays volleyball. So something plays volleyball.\nA: Let's think step by step.\
\ So something plays volleyball. (D) Marina is a dancer. No weakling is a dancer.\ \ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ Either everything is a weakling or Georgia plays volleyball. So something plays\ \ by step. Let “D” denote “being a dancer”, “m” denote “Maria”, “g” denote “Georgia”,\
\ volleyball.\nA: Let's think step by step. We refer to Wikipedia articles on formal\ \ “W” denote “weakling”, “A” denote “playing volleyball”. Then, we have the following:\n\
\ logic for help. Let\u2019s solve this step by step. Let \u201CD\u201D denote \u201C\ 1. Dm → Maria is a dance.\n2. (∀x)(Wx ⊃ ~Dx). → For all x, if x is a weakling, then\
being a dancer\u201D, \u201Cm\u201D denote \u201CMaria\u201D, \u201Cg\u201D denote\ \ x is not a dancer. In other words, no weakling is a dancer.\n3. (∀x)Wx ∨ Ag\t\
\ \u201CGeorgia\u201D, \u201CW\u201D denote \u201Cweakling\u201D, \u201CA\u201D\ / (∃x)Ax → For all x, x is a weakling or Georgia plays volleyball. So there exists\
\ denote \u201Cplaying volleyball\u201D. Then, we have the following:\n1. Dm \u2192\ \ an x that plays volleyball. \nOptions (A) and (C) do claim that some weaklings\
\ Maria is a dance.\n2. (\u2200x)(Wx \u2283 ~Dx). \u2192 For all x, if x is a weakling,\ \ are not dancers, but the second argument strongly states that no weakling is a\
\ then x is not a dancer. In other words, no weakling is a dancer.\n3. (\u2200x)Wx\ \ dancer. Thus, we can eliminate them. Option (B) omits the important detail about\
\ \u2228 Ag\t/ (\u2203x)Ax \u2192 For all x, x is a weakling or Georgia plays volleyball.\ \ Georgia playing volleyball. Option (D) has all the details presented in the arguments\
\ So there exists an x that plays volleyball. \nOptions (A) and (C) do claim that\ \ and is the best English interpretation of the arguments. The answer is (D).\n\n\
\ some weaklings are not dancers, but the second argument strongly states that no\ Q: Select the best translation into predicate logic: No people drive on Mars.\n\
\ weakling is a dancer. Thus, we can eliminate them. Option (B) omits the important\ (A) ~Pd (B) (∀x)(Px ∨ ~Dx) (C) (∀x)(Px ⊃ ~Dx) (D) ~Dp\nA: Let's think step by step.\
\ detail about Georgia playing volleyball. Option (D) has all the details presented\ \ We refer to Wikipedia articles on formal logic for help. Let’s solve this step\
\ in the arguments and is the best English interpretation of the arguments. The\ \ by step. Let “P” denote “being on Mars” and “D” denote “driving on Mars”. Then\
\ answer is (D).\n\nQ: Select the best translation into predicate logic: No people\ \ let’s consider each option:\nOption (A): ~Pd → d is not on Mars.\nOption (B):\
\ drive on Mars.\n(A) ~Pd (B) (\u2200x)(Px \u2228 ~Dx) (C) (\u2200x)(Px \u2283 ~Dx)\ \ (∀x)(Px ∨ ~Dx) → For all x, x is on Mars and x do not drive on Mars.\nOption (C):\
\ (D) ~Dp\nA: Let's think step by step. We refer to Wikipedia articles on formal\ \ (∀x)(Px ⊃ ~Dx) → For all x, x is on Mars implies that x do not drive on Mars.\n\
\ logic for help. Let\u2019s solve this step by step. Let \u201CP\u201D denote \u201C\ Option (D): ~Dp: → p do not drive on Mars.\nOf all these options, Option (C) appears\
being on Mars\u201D and \u201CD\u201D denote \u201Cdriving on Mars\u201D. Then let\u2019\ \ to be the best and most meaningful interpretation of the argument “No people drive\
s consider each option:\nOption (A): ~Pd \u2192 d is not on Mars.\nOption (B): (\u2200\ \ on Mars.” The answer is (C)."
x)(Px \u2228 ~Dx) \u2192 For all x, x is on Mars and x do not drive on Mars.\nOption\ "group": "mmlu_flan_cot_fewshot_humanities"
\ (C): (\u2200x)(Px \u2283 ~Dx) \u2192 For all x, x is on Mars implies that x do\ "include": "_mmlu_flan_cot_fewshot_template_yaml"
\ not drive on Mars.\nOption (D): ~Dp: \u2192 p do not drive on Mars.\nOf all these\ "task": "mmlu_flan_cot_fewshot_formal_logic"
\ options, Option (C) appears to be the best and most meaningful interpretation\
\ of the argument \u201CNo people drive on Mars.\u201D The answer is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_formal_logic
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml
View file @ 2184b8de
dataset_name: global_facts "dataset_name": "global_facts"
description: "The following are multiple choice questions (with answers) about global\ "description": "The following are multiple choice questions (with answers) about global\
\ facts.\n\nQ: As of 2017, how many of the world\u2019s 1-year-old children today\ \ facts.\n\nQ: As of 2017, how many of the world’s 1-year-old children today have\
\ have been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\n\ \ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%\nA: Let's\
A: Let's think step by step. We refer to Wikipedia articles on global facts for\ \ think step by step. We refer to Wikipedia articles on global facts for help. According\
\ help. According to data published by the World Health Organization, the nummber\ \ to data published by the World Health Organization, the nummber of 1-year-old\
\ of 1-year-old children vaccinated in 2017 exceeds 80%. The answer is (A).\n\n\ \ children vaccinated in 2017 exceeds 80%. The answer is (A).\n\nQ: As of 2019,\
Q: As of 2019, about what percentage of Americans agree that the state is run for\ \ about what percentage of Americans agree that the state is run for the benefit\
\ the benefit of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think\ \ of all the people?\n(A) 31% (B) 46% (C) 61% (D) 76%\nA: Let's think step by step.\
\ step by step. We refer to Wikipedia articles on global facts for help. In 2019,\ \ We refer to Wikipedia articles on global facts for help. In 2019, about 46% percentage\
\ about 46% percentage of Americans agree that the state is run for the benefit\ \ of Americans agree that the state is run for the benefit of all the people. The\
\ of all the people. The answer is (B).\n\nQ: As of 2019, about what percentage\ \ answer is (B).\n\nQ: As of 2019, about what percentage of Russians say it is very\
\ of Russians say it is very important to have free media in our country without\ \ important to have free media in our country without government/state censorship?\n\
\ government/state censorship?\n(A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think\ (A) 38% (B) 53% (C) 68% (D) 83%\nA: Let's think step by step. We refer to Wikipedia\
\ step by step. We refer to Wikipedia articles on global facts for help. As of 2019,\ \ articles on global facts for help. As of 2019, about 38% of Russians say it is\
\ about 38% of Russians say it is very important to have free media in our country.\ \ very important to have free media in our country. The answer is (A).\n\nQ: As\
\ The answer is (A).\n\nQ: As of 2015, since 1990 forests have ____ in Europe and\ \ of 2015, since 1990 forests have ____ in Europe and have ____ in Africa and the\
\ have ____ in Africa and the Americas.\n(A) increased, increased (B) increased,\ \ Americas.\n(A) increased, increased (B) increased, decreased (C) decreased, increased\
\ decreased (C) decreased, increased (D) decreased, decreased\nA: Let's think step\ \ (D) decreased, decreased\nA: Let's think step by step. We refer to Wikipedia articles\
\ by step. We refer to Wikipedia articles on global facts for help. As of 2015,\ \ on global facts for help. As of 2015, since 1990 forests have increased in Europe\
\ since 1990 forests have increased in Europe and have decreased in Africa and the\ \ and have decreased in Africa and the Americas. The answer is (B).\n\nQ: Which\
\ Americas. The answer is (B).\n\nQ: Which of the following pairs of statements\ \ of the following pairs of statements are both true (as of 2019)?\n(A) People tend\
\ are both true (as of 2019)?\n(A) People tend to be optimistic about their own\ \ to be optimistic about their own future and the future of their nation or the\
\ future and the future of their nation or the world. (B) People tend to be optimistic\ \ world. (B) People tend to be optimistic about their own future but pessimistic\
\ about the future of their nation or the world. (C) People tend to be pessimistic\
\ about their own future but optimistic about the future of their nation or the\
\ world. (D) People tend to be pessimistic about their own future and the future\
\ of their nation or the world.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on global facts for help. As of 2019, most people tend to be optimistic\
\ about their own future but pessimistic about the future of their nation or the\ \ about their own future but pessimistic about the future of their nation or the\
\ world. (C) People tend to be pessimistic about their own future but optimistic\ \ world. The answer is (B)."
\ about the future of their nation or the world. (D) People tend to be pessimistic\ "group": "mmlu_flan_cot_fewshot_other"
\ about their own future and the future of their nation or the world.\nA: Let's\ "include": "_mmlu_flan_cot_fewshot_template_yaml"
\ think step by step. We refer to Wikipedia articles on global facts for help. As\ "task": "mmlu_flan_cot_fewshot_global_facts"
\ of 2019, most people tend to be optimistic about their own future but pessimistic\
\ about the future of their nation or the world. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_global_facts
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml
View file @ 2184b8de
dataset_name: high_school_biology "dataset_name": "high_school_biology"
description: "The following are multiple choice questions (with answers) about high\ "description": "The following are multiple choice questions (with answers) about high\
\ school biology.\n\nQ: In animal cells, which of the following represents the most\ \ school biology.\n\nQ: In animal cells, which of the following represents the most\
\ likely pathway that a secretory protein takes as it is synthesized in a cell?\n\ \ likely pathway that a secretory protein takes as it is synthesized in a cell?\n\
(A) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory vesicle\u2013\ (A) Plasma membrane–Golgi apparatus–ribosome–secretory vesicle–rough ER (B) Ribosome–Golgi\
rough ER (B) Ribosome\u2013Golgi apparatus\u2013rough ER\u2013secretory vesicle\u2013\ \ apparatus–rough ER–secretory vesicle–plasma membrane (C) Plasma membrane–Golgi\
plasma membrane (C) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory\ \ apparatus–ribosome–secretory vesicle–rough ER (D) Ribosome–rough ER–Golgi apparatus–secretory\
\ vesicle\u2013rough ER (D) Ribosome\u2013rough ER\u2013Golgi apparatus\u2013secretory\ \ vesicle–plasma membrane\nA: Let's think step by step. Protein synthesis starts\
\ vesicle\u2013plasma membrane\nA: Let's think step by step. Protein synthesis starts\
\ at the ribosome, so we can eliminate (A) and (C). The ribosome is often in the\ \ at the ribosome, so we can eliminate (A) and (C). The ribosome is often in the\
\ endoplasmic reticulum and moves from there to the Golgi apparatus, where it is\ \ endoplasmic reticulum and moves from there to the Golgi apparatus, where it is\
\ modified and packaged into a vesicle. The vesicle then floats to the plasma membrane\ \ modified and packaged into a vesicle. The vesicle then floats to the plasma membrane\
\ and is secreted. The answer is (D).\n\nQ: A mutation in a bacterial enzyme changed\ \ and is secreted. The answer is (D).\n\nQ: A mutation in a bacterial enzyme changed\
\ a previously polar amino acid into a nonpolar amino acid. This amino acid was\ \ a previously polar amino acid into a nonpolar amino acid. This amino acid was\
\ located at a site distant from the enzyme\u2019s active site. How might this mutation\ \ located at a site distant from the enzyme’s active site. How might this mutation\
\ alter the enzyme\u2019s substrate specificity?\n(A) By changing the enzyme\u2019\ \ alter the enzyme’s substrate specificity?\n(A) By changing the enzyme’s pH optimum\
s pH optimum (B) By changing the enzyme\u2019s location in the cell (C) By changing\ \ (B) By changing the enzyme’s location in the cell (C) By changing the shape of\
\ the shape of the protein (D) An amino acid change away from the active site cannot\ \ the protein (D) An amino acid change away from the active site cannot alter the\
\ alter the enzyme\u2019s substrate specificity.\nA: Let's think step by step. A\ \ enzyme’s substrate specificity.\nA: Let's think step by step. A change in an amino\
\ change in an amino acid leads to a change in the primary structure of the protein.\ \ acid leads to a change in the primary structure of the protein. A change in the\
\ A change in the primary structure may lead to a change in the secondary and the\ \ primary structure may lead to a change in the secondary and the tertiary structure\
\ tertiary structure of the protein. A change in the tertiary structure means a\ \ of the protein. A change in the tertiary structure means a change in the shape\
\ change in the shape of the protein, so (C) has to be correct. Since the change\ \ of the protein, so (C) has to be correct. Since the change does not affect the\
\ does not affect the active site of the enzyme, we do not expect the activity of\ \ active site of the enzyme, we do not expect the activity of the enzyme to be affected.\
\ the enzyme to be affected. The answer is (C).\n\nQ: Which of the following is\ \ The answer is (C).\n\nQ: Which of the following is not a way to form recombinant\
\ not a way to form recombinant DNA?\n(A) Translation (B) Conjugation (C) Specialized\ \ DNA?\n(A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation\n\
\ transduction (D) Transformation\nA: Let's think step by step. The introduction\ A: Let's think step by step. The introduction of foreign DNA or RNA into bacteria\
\ of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique\ \ or eukaryotic cells is a common technique in molecular biology and scientific\
\ in molecular biology and scientific research. There are multiple ways foreign\ \ research. There are multiple ways foreign DNA can be introduced into cells including\
\ DNA can be introduced into cells including transformation, transduction, conjugation,\ \ transformation, transduction, conjugation, and transfection. In contrast, (A)\
\ and transfection. In contrast, (A) is not a way to form DNA: during translation\ \ is not a way to form DNA: during translation the ribosomes synthesize proteins\
\ the ribosomes synthesize proteins from RNA. The answer is (A).\n\nQ: Homologous\ \ from RNA. The answer is (A).\n\nQ: Homologous structures are often cited as evidence\
\ structures are often cited as evidence for the process of natural selection. All\ \ for the process of natural selection. All of the following are examples of homologous\
\ of the following are examples of homologous structures EXCEPT\n(A) the wings of\ \ structures EXCEPT\n(A) the wings of a bird and the wings of a bat (B) the flippers\
\ a bird and the wings of a bat (B) the flippers of a whale and the arms of a man\ \ of a whale and the arms of a man (C) the pectoral fins of a porpoise and the flippers\
\ (C) the pectoral fins of a porpoise and the flippers of a seal (D) the forelegs\ \ of a seal (D) the forelegs of an insect and the forelimbs of a dog\nA: Let's think\
\ of an insect and the forelimbs of a dog\nA: Let's think step by step. \u200B\u200B\ \ step by step. ​​Homologous structures are similar physical features in organisms\
Homologous structures are similar physical features in organisms that share a common\ \ that share a common ancestor ​​but different functions. Comparisons (B) and (C)\
\ ancestor \u200B\u200Bbut different functions. Comparisons (B) and (C) are clearly\ \ are clearly homologous because they share a common ancestor and the structures\
\ homologous because they share a common ancestor and the structures serve different\ \ serve different purposes. Bat wings and birg wings are also homologous, while\
\ purposes. Bat wings and birg wings are also homologous, while they are both wings,\ \ they are both wings, the forelimbs serve different purposes. Insects and dogs\
\ the forelimbs serve different purposes. Insects and dogs are very far ancestors\ \ are very far ancestors since one is vertebrate while the other is invertebrate\
\ since one is vertebrate while the other is invertebrate and the forelimbs serve\ \ and the forelimbs serve the same purpose, so they are not homologous. The answer\
\ the same purpose, so they are not homologous. The answer is (D).\n\nQ: Which of\ \ is (D).\n\nQ: Which of the following is not known to be involved in the control\
\ the following is not known to be involved in the control of cell division?\n(A)\ \ of cell division?\n(A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast\
\ Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast cells\nA: Let's think\ \ cells\nA: Let's think step by step. Normal cells move through the cell cycle in\
\ step by step. Normal cells move through the cell cycle in a regulated way. At\ \ a regulated way. At the checkpoint stage, they use information about their own\
\ the checkpoint stage, they use information about their own internal state and\ \ internal state and cues from the environment around them to decide whether to\
\ cues from the environment around them to decide whether to proceed with cell division.\ \ proceed with cell division. Cues like these act by changing the activity of core\
\ Cues like these act by changing the activity of core cell cycle regulators inside\ \ cell cycle regulators inside the cell. The most common regulators are cyclins\
\ the cell. The most common regulators are cyclins and cyclin-dependent kinases.\ \ and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division.\
\ Fibroblast cells do not play any role in cell division. The answer is (D)." \ The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_high_school_biology "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_biology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_chemistry.yaml
View file @ 2184b8de
dataset_name: high_school_chemistry "dataset_name": "high_school_chemistry"
description: "The following are multiple choice questions (with answers) about high\ "description": "The following are multiple choice questions (with answers) about high\
\ school chemistry.\n\nQ: Which of the following is considered an acid anhydride?\n\ \ school chemistry.\n\nQ: Which of the following is considered an acid anhydride?\n\
(A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3\nA: Let's think step by step. An acid anhydride\ (A) HCl (B) H2SO3 (C) SO2 (D) Al(NO3)3\nA: Let's think step by step. An acid anhydride\
\ is a compound that is derived by removing water from an acid. The chemical formula\ \ is a compound that is derived by removing water from an acid. The chemical formula\
...@@ -45,5 +45,6 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -45,5 +45,6 @@ description: "The following are multiple choice questions (with answers) about h
\ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\ \ the acetate ion. The added strong acid, Nitric acid, will react with the conjugate\
\ base. Therefore the maximum amount of acid that can be added will be equal to\ \ base. Therefore the maximum amount of acid that can be added will be equal to\
\ the amount of acetate ion, or 2 moles. The answer is (C)." \ the amount of acetate ion, or 2 moles. The answer is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_high_school_chemistry "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_chemistry"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml
View file @ 2184b8de
dataset_name: high_school_computer_science "dataset_name": "high_school_computer_science"
description: "The following are multiple choice questions (with answers) about high\ "description": "The following are multiple choice questions (with answers) about high\
\ school computer science.\n\nQ: Which of the following is an example of the use\ \ school computer science.\n\nQ: Which of the following is an example of the use\
\ of a device on the Internet of Things (IoT) ?\n(A) A car alerts a driver that\ \ of a device on the Internet of Things (IoT) ?\n(A) A car alerts a driver that\
\ it is about to hit an object. (B) A hiker uses a G P S watch to keep track of\ \ it is about to hit an object. (B) A hiker uses a G P S watch to keep track of\
...@@ -26,9 +26,9 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -26,9 +26,9 @@ description: "The following are multiple choice questions (with answers) about h
\ launched from any web sites visited or files downloaded.\nA: Let's think step\ \ launched from any web sites visited or files downloaded.\nA: Let's think step\
\ by step. Choice A is incorrect as it only describes network traffic, which an\ \ by step. Choice A is incorrect as it only describes network traffic, which an\
\ anonymous browser does not change. Choice B is correct as it correctly describes\ \ anonymous browser does not change. Choice B is correct as it correctly describes\
\ how an anonymous browser will prevent saving data on the user\u2019s computer\ \ how an anonymous browser will prevent saving data on the user’s computer after\
\ after the session is ended. Choice C is incorrect because an anonymous browser\ \ the session is ended. Choice C is incorrect because an anonymous browser will\
\ will not prevent logging in to email or social media accounts. Choice D is incorrect\ \ not prevent logging in to email or social media accounts. Choice D is incorrect\
\ because an anonymous browser in itself performs no virus protection. The answer\ \ because an anonymous browser in itself performs no virus protection. The answer\
\ is (B).\n\nQ: In the program below, the initial value of X is 5 and the initial\ \ is (B).\n\nQ: In the program below, the initial value of X is 5 and the initial\
\ value of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\ \ value of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\
...@@ -66,5 +66,6 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -66,5 +66,6 @@ description: "The following are multiple choice questions (with answers) about h
\ its value is greater than 100, regardless of the elements in the list. Choice\ \ its value is greater than 100, regardless of the elements in the list. Choice\
\ D is incorrect because its step 3 does not increment the value of position, so\ \ D is incorrect because its step 3 does not increment the value of position, so\
\ it will repeat forever. The answer is (B)." \ it will repeat forever. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_stem"
task: mmlu_flan_cot_fewshot_high_school_computer_science "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_computer_science"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml
View file @ 2184b8de
dataset_name: high_school_european_history "dataset_name": "high_school_european_history"
description: "The following are multiple choice questions (with answers) about high\ "description": "The following are multiple choice questions (with answers) about high\
\ school european history.\n\nQ: This question refers to the following information.\n\ \ school european history.\n\nQ: This question refers to the following information.\n\
Albeit the king's Majesty justly and rightfully is and ought to be the supreme head\ Albeit the king's Majesty justly and rightfully is and ought to be the supreme head\
\ of the Church of England, and so is recognized by the clergy of this realm in\ \ of the Church of England, and so is recognized by the clergy of this realm in\
...@@ -34,7 +34,7 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -34,7 +34,7 @@ description: "The following are multiple choice questions (with answers) about h
\ the corruption in the Church of England. The answer is (D).\n\nQ: This question\ \ the corruption in the Church of England. The answer is (D).\n\nQ: This question\
\ refers to the following information.\nRead the following excerpt.\nThe revolutionary\ \ refers to the following information.\nRead the following excerpt.\nThe revolutionary\
\ seed had penetrated into every country and spread more or less. It was greatly\ \ seed had penetrated into every country and spread more or less. It was greatly\
\ developed under the r\xE9gime of the military despotism of Bonaparte. His conquests\ \ developed under the régime of the military despotism of Bonaparte. His conquests\
\ displaced a number of laws, institutions, and customs; broke through bonds sacred\ \ displaced a number of laws, institutions, and customs; broke through bonds sacred\
\ among all nations, strong enough to resist time itself; which is more than can\ \ among all nations, strong enough to resist time itself; which is more than can\
\ be said of certain benefits conferred by these innovators.\nThe monarchs will\ \ be said of certain benefits conferred by these innovators.\nThe monarchs will\
...@@ -55,9 +55,9 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -55,9 +55,9 @@ description: "The following are multiple choice questions (with answers) about h
Let them maintain religious principles in all their purity, and not allow the faith\ Let them maintain religious principles in all their purity, and not allow the faith\
\ to be attacked and morality interpreted according to the social contract or the\ \ to be attacked and morality interpreted according to the social contract or the\
\ visions of foolish sectarians.\nLet them suppress Secret Societies; that gangrene\ \ visions of foolish sectarians.\nLet them suppress Secret Societies; that gangrene\
\ of society.\n\u2014Klemens von Metternich, Political Confession of Faith, 1820\n\ \ of society.\n—Klemens von Metternich, Political Confession of Faith, 1820\nWhich\
Which of the following was the greatest cause of the fears expressed by Metternich\ \ of the following was the greatest cause of the fears expressed by Metternich in\
\ in the document above?\n(A) The ideas of personal liberty and nationalism conceived\ \ the document above?\n(A) The ideas of personal liberty and nationalism conceived\
\ during the Enlightenment resulted in radical revolutions that could spread throughout\ \ during the Enlightenment resulted in radical revolutions that could spread throughout\
\ Europe. (B) The conquest of Europe by Napoleon led to the creation of new factions\ \ Europe. (B) The conquest of Europe by Napoleon led to the creation of new factions\
\ and shifted the European balance of power. (C) The power of monarchs had grown\ \ and shifted the European balance of power. (C) The power of monarchs had grown\
...@@ -110,15 +110,15 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -110,15 +110,15 @@ description: "The following are multiple choice questions (with answers) about h
\ were all turning to the politicians; the famous Nihilists who made Europe tremble-sons\ \ were all turning to the politicians; the famous Nihilists who made Europe tremble-sons\
\ of village priests, of the lower middle class, of tradesmen-could not rise above\ \ of village priests, of the lower middle class, of tradesmen-could not rise above\
\ the idea of national liberation, and seemed to believe that the world would be\ \ the idea of national liberation, and seemed to believe that the world would be\
\ delivered-when they had killed their despot&\u2026\n\"Foolery! They'll never get\ \ delivered-when they had killed their despot&…\n\"Foolery! They'll never get out\
\ out of it with their foolery.\"\nThen, lowering his voice still more, in a few\ \ of it with their foolery.\"\nThen, lowering his voice still more, in a few bitter\
\ bitter words he described his old dream of fraternity. He had renounced his rank\ \ words he described his old dream of fraternity. He had renounced his rank and\
\ and his fortune; he had gone among workmen, only in the hope of seeing at last\ \ his fortune; he had gone among workmen, only in the hope of seeing at last the\
\ the foundation of a new society of labour in common. All the sous in his pockets\ \ foundation of a new society of labour in common. All the sous in his pockets had\
\ had long gone to the urchins of the settlement; he had been as tender as a brother\ \ long gone to the urchins of the settlement; he had been as tender as a brother\
\ with the colliers, smiling at their suspicion, winning them over by his quiet\ \ with the colliers, smiling at their suspicion, winning them over by his quiet\
\ workmanlike ways and his dislike of chattering. But decidedly the fusion had not\ \ workmanlike ways and his dislike of chattering. But decidedly the fusion had not\
\ taken place.\nHis voice changed, his eyes grew bright, he fixed them on \xE9tienne,\ \ taken place.\nHis voice changed, his eyes grew bright, he fixed them on étienne,\
\ directly addressing him:\n\"Now, do you understand that? These hatworkers at Marseilles\ \ directly addressing him:\n\"Now, do you understand that? These hatworkers at Marseilles\
\ who have won the great lottery prize of a hundred thousand francs have gone off\ \ who have won the great lottery prize of a hundred thousand francs have gone off\
\ at once and invested it, declaring that they are going to live without doing anything!\ \ at once and invested it, declaring that they are going to live without doing anything!\
...@@ -127,7 +127,7 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -127,7 +127,7 @@ description: "The following are multiple choice questions (with answers) about h
\ out as much as you like against the rich, you haven't got courage enough to give\ \ out as much as you like against the rich, you haven't got courage enough to give\
\ back to the poor the money that luck brings you. You will never be worthy of happiness\ \ back to the poor the money that luck brings you. You will never be worthy of happiness\
\ as long as you own anything, and your hatred of the bourgeois proceeds solely\ \ as long as you own anything, and your hatred of the bourgeois proceeds solely\
\ from an angry desire to be bourgeois yourselves in their place.\"\n\xE9mile Zola,\ \ from an angry desire to be bourgeois yourselves in their place.\"\némile Zola,\
\ French writer, Germinal, 1885\nThe passage displays the direct concern for the\ \ French writer, Germinal, 1885\nThe passage displays the direct concern for the\
\ welfare of the working classes that was typically a part of which movement?\n\ \ welfare of the working classes that was typically a part of which movement?\n\
(A) Capitalist (B) Scientific (C) Communist (D) Existentialist\nA: Let's think step\ (A) Capitalist (B) Scientific (C) Communist (D) Existentialist\nA: Let's think step\
...@@ -156,13 +156,14 @@ description: "The following are multiple choice questions (with answers) about h ...@@ -156,13 +156,14 @@ description: "The following are multiple choice questions (with answers) about h
\ whether Jewish, Christian or Turkish, appear to me no other than human inventions,\ \ whether Jewish, Christian or Turkish, appear to me no other than human inventions,\
\ set up to terrify and enslave mankind, and monopolize power and profit.\nI do\ \ set up to terrify and enslave mankind, and monopolize power and profit.\nI do\
\ not mean by this declaration to condemn those who believe otherwise; they have\ \ not mean by this declaration to condemn those who believe otherwise; they have\
\ the same right to their belief as I have to mine.\n\u2014Thomas Paine, The Age\ \ the same right to their belief as I have to mine.\n—Thomas Paine, The Age of Reason,\
\ of Reason, 1794\u20131795\nWhich of the following Enlightenment philosophes designed\ \ 1794–1795\nWhich of the following Enlightenment philosophes designed a system\
\ a system of checks and balances for government to avoid abuses of power?\n(A)\ \ of checks and balances for government to avoid abuses of power?\n(A) Jean Jacques\
\ Jean Jacques Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\n\ \ Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft (D) Adam Smith\nA: Let's\
A: Let's think step by step. We refer to Wikipedia articles on european history\ \ think step by step. We refer to Wikipedia articles on european history for help.\
\ for help. Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\ \ Baron Montesquieu was a 18th centrury French philsopher who wrote extensively\
\ against the monoplization of power and advocated for a system of checks and balances\ \ against the monoplization of power and advocated for a system of checks and balances\
\ in government to prevent the rise of despotism. The answer is (B)." \ in government to prevent the rise of despotism. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml "group": "mmlu_flan_cot_fewshot_humanities"
task: mmlu_flan_cot_fewshot_high_school_european_history "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_european_history"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml
View file @ 2184b8de
dataset_name: high_school_geography "dataset_name": "high_school_geography"
description: 'The following are multiple choice questions (with answers) about high "description": "The following are multiple choice questions (with answers) about high\
school geography. \ school geography.\n\nQ: Which one of the following items is an example of nonmaterial\
\ culture?\n(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).\n\
A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
Q: Which one of the following items is an example of nonmaterial culture? \ Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not\
\ physical objects. The answer is (C).\n\nQ: During the third stage of the demographic\
(A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird). \ transition model, which of the following is true?\n(A) Birth rates increase and\
\ population growth rate is less rapid. (B) Birth rates decline and population growth\
A: Let''s think step by step. We refer to Wikipedia articles on geography for help. \ rate is less rapid. (C) Birth rates increase and population growth rate increases.\
Nonmaterial culture consists of cultural ideas, beliefs or symbols that are not \ (D) Birth rates decrease and population growth rate increases.\nA: Let's think\
physical objects. The answer is (C). \ step by step. We refer to Wikipedia articles on geography for help. The demographic\
\ transition model models the five different stages of population growth as a country\
\ goes through economic development, where the third stage refers to a period of\
Q: During the third stage of the demographic transition model, which of the following \ declining birth rates and lower population growth. The answer is (B).\n\nQ: The\
is true? \ practice of hiring a foreign third-party service provider to run an operation\
\ is called\n(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.\n\
(A) Birth rates increase and population growth rate is less rapid. (B) Birth rates A: Let's think step by step. We refer to Wikipedia articles on geography for help.\
decline and population growth rate is less rapid. (C) Birth rates increase and population \ \"Offshoring\" literally means to move or base some of the activities or processes\
growth rate increases. (D) Birth rates decrease and population growth rate increases. \ of a company to a foreign country. The answer is (B).\n\nQ: Which of the following\
\ statements is NOT accurate regarding the services provided by local governments\
A: Let''s think step by step. We refer to Wikipedia articles on geography for help. \ in the United States?\n(A) Duplication of efforts occurs often. (B) Social problems\
The demographic transition model models the five different stages of population \ of the central city spill over into the surrounding residential suburbs. (C) Inefficiency\
growth as a country goes through economic development, where the third stage refers \ in providing services occurs often. (D) One neighborhood's efforts to reduce pollution\
to a period of declining birth rates and lower population growth. The answer is \ are always supported by neighboring communities.\nA: Let's think step by step.\
(B). \ We refer to Wikipedia articles on geography for help. There may be economic, social\
\ or political reasons for two neighboring communities and their local governments\
\ not agreeing to pollution reduction efforts initiated by one of them. The answer\
Q: The practice of hiring a foreign third-party service provider to run an operation \ is (D).\n\nQ: The rate of natural increase of a population is found by subtracting\
is called \ the\n(A) crude death rate from the crude birth date. (B) crude birth rate from\
\ the crude death rate. (C) doubling time from the crude birth rate. (D) fertility\
(A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence. \ rate from the crude death rate.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on geography for help. The difference between number of births and deaths\
A: Let''s think step by step. We refer to Wikipedia articles on geography for help. \ gives the population increase at any given time. The answer is (A)."
"Offshoring" literally means to move or base some of the activities or processes "group": "mmlu_flan_cot_fewshot_social_sciences"
of a company to a foreign country. The answer is (B). "include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_high_school_geography"
Q: Which of the following statements is NOT accurate regarding the services provided
by local governments in the United States?
(A) Duplication of efforts occurs often. (B) Social problems of the central city
spill over into the surrounding residential suburbs. (C) Inefficiency in providing
services occurs often. (D) One neighborhood''s efforts to reduce pollution are always
supported by neighboring communities.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
There may be economic, social or political reasons for two neighboring communities
and their local governments not agreeing to pollution reduction efforts initiated
by one of them. The answer is (D).
Q: The rate of natural increase of a population is found by subtracting the
(A) crude death rate from the crude birth date. (B) crude birth rate from the crude
death rate. (C) doubling time from the crude birth rate. (D) fertility rate from
the crude death rate.
A: Let''s think step by step. We refer to Wikipedia articles on geography for help.
The difference between number of births and deaths gives the population increase
at any given time. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_geography
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml
View file @ 2184b8de
dataset_name: high_school_government_and_politics "dataset_name": "high_school_government_and_politics"
description: 'The following are multiple choice questions (with answers) about high "description": "The following are multiple choice questions (with answers) about high\
school government and politics. \ school government and politics.\n\nQ: Which of the following best states an argument\
\ made by James Madison in The Federalist number 10?\n(A) Honest politicians can\
\ prevent factions from developing. (B) Factions are more likely to occur in large\
Q: Which of the following best states an argument made by James Madison in The Federalist \ republics than in small ones. (C) The negative effects of factionalism can be\
number 10? \ reduced by a republican government. (D) Free elections are the people's best defense\
\ against factionalism.\nA: Let's think step by step. We refer to Wikipedia articles\
(A) Honest politicians can prevent factions from developing. (B) Factions are more \ on government and politics for help. In the Federalist number 10, James Madison\
likely to occur in large republics than in small ones. (C) The negative effects \ advocated for a representative republican form of government to guard against\
of factionalism can be reduced by a republican government. (D) Free elections are \ factionalism. The answer is (C).\n\nQ: The term \"budget deficit\" refers to the\n\
the people''s best defense against factionalism. (A) annual increase in federal spending on the military (B) amount of interest on\
\ the national debt (C) difference between the initial budget proposals made by\
A: Let''s think step by step. We refer to Wikipedia articles on government and politics \ the president and Congress (D) amount the government spends in excess of its revenues\n\
for help. In the Federalist number 10, James Madison advocated for a representative A: Let's think step by step. We refer to Wikipedia articles on government and politics\
republican form of government to guard against factionalism. The answer is (C). \ for help. When the goverment spends more than it earns, their difference is the\
\ budget deficit. The answer is (D).\n\nQ: Which of the following statements about\
\ cabinet departments is FALSE?\n(A) They are established by the legislative branch.\
Q: The term "budget deficit" refers to the \ (B) Their members often don't have much influence over presidential decisions.\
\ (C) They cannot all be run by leaders who belong to the same political party the\
(A) annual increase in federal spending on the military (B) amount of interest on \ president does. (D) Not every federal agency is a cabinet department.\nA: Let's\
the national debt (C) difference between the initial budget proposals made by the \ think step by step. We refer to Wikipedia articles on government and politics\
president and Congress (D) amount the government spends in excess of its revenues \ for help. There is no law stipulating that some cabinet department leaders have\
\ to belong to a political party different from that of the president. The answer\
A: Let''s think step by step. We refer to Wikipedia articles on government and politics \ is (C).\n\nQ: Which of the following cases established the precedent that a defendant\
for help. When the goverment spends more than it earns, their difference is the \ must be informed of the right to remain silent, the right to a lawyer, and protection\
budget deficit. The answer is (D). \ from self-incrimination?\n(A) Weeks v. United States (B) Betts v. Brady (C) Mapp\
\ v. Ohio (D) Miranda v. Arizona\nA: Let's think step by step. We refer to Wikipedia\
\ articles on government and politics for help. In the landmark Miranda v. Arizona\
Q: Which of the following statements about cabinet departments is FALSE? \ in 1966, the US Supreme Court, based on the Fifth and Sixth Amendment of the US\
\ Constitution, guaranteed a defendant's right to an attorney and protection from\
(A) They are established by the legislative branch. (B) Their members often don''t \ self-incrimination. The answer is (D).\n\nQ: Uncertainty over the limits to presidential\
have much influence over presidential decisions. (C) They cannot all be run by leaders \ power is caused primarily by the fact that\n(A) the constitutional definition\
who belong to the same political party the president does. (D) Not every federal \ of those powers is broad and unspecific (B) most people agree that the Constitution\
agency is a cabinet department. \ places too many limits on presidential power (C) the Supreme Court consistently\
\ refuses to rule on cases concerning presidential powers (D) constitutional amendments\
A: Let''s think step by step. We refer to Wikipedia articles on government and politics \ have greatly increased presidential powers\nA: Let's think step by step. We refer\
for help. There is no law stipulating that some cabinet department leaders have \ to Wikipedia articles on government and politics for help. The US Constitution\
to belong to a political party different from that of the president. The answer \ is not very specific about the powers of the president, leading to uncertainty\
is (C). \ over its limits. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
Q: Which of the following cases established the precedent that a defendant must "task": "mmlu_flan_cot_fewshot_high_school_government_and_politics"
be informed of the right to remain silent, the right to a lawyer, and protection
from self-incrimination?
(A) Weeks v. United States (B) Betts v. Brady (C) Mapp v. Ohio (D) Miranda v. Arizona
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. In the landmark Miranda v. Arizona in 1966, the US Supreme Court, based
on the Fifth and Sixth Amendment of the US Constitution, guaranteed a defendant''s
right to an attorney and protection from self-incrimination. The answer is (D).
Q: Uncertainty over the limits to presidential power is caused primarily by the
fact that
(A) the constitutional definition of those powers is broad and unspecific (B) most
people agree that the Constitution places too many limits on presidential power
(C) the Supreme Court consistently refuses to rule on cases concerning presidential
powers (D) constitutional amendments have greatly increased presidential powers
A: Let''s think step by step. We refer to Wikipedia articles on government and politics
for help. The US Constitution is not very specific about the powers of the president,
leading to uncertainty over its limits. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_high_school_government_and_politics
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment