merged latest update

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml

+dataset_name: professional_law
+description: "The following are multiple choice questions (with answers) about professional\
+  \ law.\n\nQ: A son owed a creditor $5,000. The son's father contacted the creditor\
+  \ and told him that he wanted to pay the son's debt. The father signed a document\
+  \ that stated the father would pay the son's debt at a rate of $500 a month for\
+  \ 10 months. The creditor made no written or oral commitment to forbear to sue the\
+  \ son to collect the $5,000 debt, and the father made no oral or written request\
+  \ for any such forbearance. For the next five months, the father made and the creditor\
+  \ accepted the $500 monthly payments as agreed. During that period, the creditor,\
+  \ in fact, did forbear to take any legal action against the son. However, the father\
+  \ then informed the creditor that he would make no further payments on the debt.\
+  \ Which of the following is the most persuasive argument that the father is liable\
+  \ to the creditor under the terms of their agreement?\n(A) The father's promise\
+  \ and the creditor's reliance thereon, if proved, gave rise to a valid claim by\
+  \ the creditor against the father based on the doctrine of promissory estoppel.\
+  \ (B) Because it was foreseeable that the father's promise would induce the creditor\
+  \ to forbear taking any action against the son, such forbearance was, as a matter\
+  \ of law, a bargained-for consideration for the father's promise. (C) The father's\
+  \ five payments to the creditor totaling $2,500 manifested a serious intent on the\
+  \ father's part to be contractually bound, and such manifestation is generally recognized\
+  \ as an effective substitute for consideration. (D) By assuming the antecedent debt\
+  \ obligation that the son owed to the creditor, the father became a surety whose\
+  \ promise to the creditor was enforceable, since it was in writing and supported\
+  \ by adequate consideration. \nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on law for help. The doctrine of promissory estoppel stops a person from\
+  \ going back on a promise in contract law, hence option (A) should be the most persuasive\
+  \ argument. The answer is (A).\n\nQ: A state has recently enacted a statute prohibiting\
+  \ the disposal of any nuclear wastes within the state. This law does not contravene\
+  \ or conflict with any federal statutes. A man operates a company in the state that\
+  \ is engaged in the disposal of nuclear wastes. Subsequent to the passage of the\
+  \ state statute, the man, not yet aware of the new law, entered into contracts with\
+  \ many out-of-state firms to dispose of their nuclear wastes in the state. On account\
+  \ of this new law, however, the man will be unable to perform these contracts. Assume\
+  \ that the man has standing to challenge this state law. Which of the following\
+  \ presents his strongest constitutional grounds to challenge the state law prohibiting\
+  \ the disposal of nuclear wastes within the state?\n(A) The commerce clause. (B)\
+  \ The equal protection clause of the Fourteenth Amendment. (C) The privileges and\
+  \ immunities clause of Article IV, Section 2. (D) The contract clause.\nA: Let's\
+  \ think step by step. We refer to Wikipedia articles on law for help. The commerce\
+  \ clause states that Congress shall have the power to regulate commerce with foreign\
+  \ Nations, and among the several States, and with the Indian Tribes. The statute\
+  \ affects inter-state commerce which puts it into question. Hence the man's strongest\
+  \ argument should be the commerce clause. The answer is (A).\n\nQ: On October 1,\
+  \ 1980, a developer, owner of several hundred acres in a rural county, drafted a\
+  \ general development plan for the area. The duly recorded plan imposed elaborate\
+  \ limitations and restrictions upon the land in the plan, which was to be developed\
+  \ as a residential district. The restrictions were to extend to all persons acquiring\
+  \ any of the lots and to their heirs, assigns, and lessees. It was further provided\
+  \ that all subsequent owners would be charged with due notice of the restrictions.\
+  \ Among those restrictions in the general plan were the following:(22) A franchise\
+  \ right is created in a strip of land 10 feet in width along the rear of each lot\
+  \ for the use of public utility companies with right of ingress and egress. (23)\
+  \ No house or structure of any kind shall be built on the aforementioned strip of\
+  \ land running through the said blocks. In 2000, a retiree purchased one of the\
+  \ lots, built a house, and erected a fence in the rear of his property within the\
+  \ restricted area. In 2004, a teacher purchased a lot adjacent to the retiree's\
+  \ property and built a new house. Two years later, a librarian purchased the lot\
+  \ that adjoined the teacher's property. The three deeds to those properties each\
+  \ contained references to the deed book where the general plan was recorded. In\
+  \ 2008, the librarian began the construction of a seven-foot post-and-rail fence\
+  \ along the line dividing his lot with the teacher's, and along the center of the\
+  \ area subject to the franchise right. Although the teacher objected to its construction,\
+  \ the fence was completed. If the teacher seeks a mandatory injunction to compel\
+  \ removal of the librarian's fence, the court will most likely\n(A) grant relief,\
+  \ because the fence was in violation of the easement restriction. (B) grant relief,\
+  \ because the encroachment of the fence violated the restriction in the original\
+  \ plan. (C) deny relief, because the teacher failed to enforce the restriction against\
+  \ the retiree. (D) deny relief, because the fence would not be construed as \"a\
+  \ structure\" within the terms of the restriction. \nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on law for help. The restrictions in the original\
+  \ plan say no house or structure of any kind shall be built on the aforementioned\
+  \ strip of land running through the said blocks. Hence the court will most likely\
+  \ grant relief because the fence violated the restriction in the original plan.\
+  \ The answer is (B).\n\nQ: Judge took judicial notice of some facts at the beginning\
+  \ of the trial. Which of the following is not an appropriate kind of fact for judicial\
+  \ notice?\n(A) Indisputable facts. (B) Facts that have been asserted by individual\
+  \ political organizations. (C) Facts recognized to be true by common knowledge.\
+  \ (D) Facts capable of scientific verification.\nA: Let's think step by step. We\
+  \ refer to Wikipedia articles on law for help. Among the options, facts that have\
+  \ been asserted by individual political organizations is not an appropriate kind\
+  \ of fact for judicial notice. The answer is (B).\n\nQ: A state legislature has\
+  \ recently enacted a statute making it a misdemeanor to curse or revile or use obscene\
+  \ or opprobrious language toward or in reference to a police officer perfonning\
+  \ his duties. A student at a state university organized a demonstration on campus\
+  \ to protest the war. The rally was attended by a group of 50 students who shouted\
+  \ anti-war messages at cars passing by. To show his contempt for the United States,\
+  \ the student sewed the American flag to the rear of his jeans. When a police officer\
+  \ saw the flag sown on the student's jeans, he approached and told him to remove\
+  \ the flag or he would be placed under arrest. The student became angered and shouted\
+  \ at the police officer, \"Listen, you bastard, I'll wear this rag anywhere I please.\
+  \ \" The student was subsequently placed under arrest and charged with violating\
+  \ the state statute. The student subsequently brings suit in state court challenging\
+  \ the constitutionality of the statute. The strongest constitutional argument for\
+  \ the student is that\n(A) the statute is void for vagueness under the Fourteenth\
+  \ Amendment's due process clause. (B) the statute is invalid because it violates\
+  \ the petitioner's freedom of speech under the First Amendment. (C) the statute\
+  \ is an abridgment of freedom of speech under the First Amendment because less restrictive\
+  \ means are available for achieving the same purpose. (D) the statute is overbroad\
+  \ and consequently invalid under the First and Fourteenth Amendments.\nA: Let's\
+  \ think step by step. We refer to Wikipedia articles on law for help. The Fourteenth\
+  \ Amendment further supports the First Amendment by establishing a due process clause.\
+  \ Hence the strongest argument should be the statute is overbroad and consequently\
+  \ invalid under the First and Fourteenth Amendments. The answer is (D)."
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_professional_law

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml

+dataset_name: professional_medicine
+description: "The following are multiple choice questions (with answers) about professional\
+  \ medicine.\n\nQ: A 22-year-old male marathon runner presents to the office with\
+  \ the complaint of right-sided rib pain when he runs long distances. Physical examination\
+  \ reveals normal heart and lung findings and an exhalation dysfunction at ribs\_\
+  4-5 on the right. Which of the following muscles or muscle groups will be most useful\
+  \ in correcting this dysfunction utilizing a direct method?\n(A) anterior scalene\
+  \ (B) latissimus dorsi (C) pectoralis minor (D) quadratus lumborum\nA: Let's think\
+  \ step by step. We refer to Wikipedia articles on medicine for help. Among the options,\
+  \ only pectoralis minor muscle origins from the outer surfaces of the 3rd to 5th\
+  \ ribs. The answer is (C).\n\nQ: A 36-year-old male presents to the office with\
+  \ a\_3-week\_history of low back pain. He denies any recent trauma but says that\
+  \ he climbs in and out of his truck numerous times a day for his job. Examination\
+  \ of the patient in the prone position reveals a deep sacral sulcus on the left,\
+  \ a posterior inferior lateral angle on the right, and a lumbosacral junction that\
+  \ springs freely on compression. The most likely diagnosis is\n(A) left-on-left\
+  \ sacral torsion (B) left-on-right sacral torsion (C) right unilateral sacral flexion\
+  \ (D) right-on-right sacral torsion\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on medicine for help. The deep sulcus on the left, a posterior ILA on\
+  \ the right, with a negative spring test suggests a right-on-right sacral torsion.\
+  \ All other options have a deep sulcus on the right. The answer is (D).\n\nQ: A\
+  \ 44-year-old man comes to the office because of a 3-day history of sore throat,\
+  \ nonproductive cough, runny nose, and frontal headache. He says the headache is\
+  \ worse in the morning and ibuprofen does provide some relief. He has not had shortness\
+  \ of breath. Medical history is unremarkable. He takes no medications other than\
+  \ the ibuprofen for pain. Vital signs are temperature 37.4\xB0C (99.4\xB0F), pulse\
+  \ 88/min, respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the\
+  \ nares shows erythematous mucous membranes. Examination of the throat shows erythema\
+  \ and follicular lymphoid hyperplasia on the posterior oropharynx. There is no palpable\
+  \ cervical adenopathy. Lungs are clear to auscultation. Which of the following is\
+  \ the most likely cause of this patient's symptoms?\n(A) Allergic rhinitis (B) Epstein-Barr\
+  \ virus (C) Mycoplasma pneumonia (D) Rhinovirus\nA: Let's think step by step. We\
+  \ refer to Wikipedia articles on medicine for help. The symptoms, especially the\
+  \ headache, suggest that the most likely cause is Rhinovirus. Epstein-Barr virus\
+  \ will cause swollen lymph nodes but there is no palpable cervical adenopathy. Lungs\
+  \ are clear to auscultation suggests it's not Mycoplasma pneumonia. The answer is\
+  \ (D).\n\nQ: A previously healthy 32-year-old woman comes to the physician 8 months\
+  \ after her husband was killed in a car crash. Since that time, she has had a decreased\
+  \ appetite and difficulty falling asleep. She states that she is often sad and cries\
+  \ frequently. She has been rechecking the door lock five times before leaving her\
+  \ house and has to count exactly five pieces of toilet paper before she uses it.\
+  \ She says that she has always been a perfectionist but these urges and rituals\
+  \ are new. Pharmacotherapy should be targeted to which of the following neurotransmitters?\n\
+  (A) Dopamine (B) Glutamate (C) Norepinephrine (D) Serotonin\nA: Let's think step\
+  \ by step. We refer to Wikipedia articles on medicine for help. The patient feels\
+  \ sad and among the options, only Dopamine and Serotonin can help increase positive\
+  \ emotions. Serotonin also affects digestion and metabolism, which can help the\
+  \ patient's decreased appetite and sleep difficulty. The answer is (D).\n\nQ: A\
+  \ 42-year-old man comes to the office for preoperative evaluation prior to undergoing\
+  \ adrenalectomy scheduled in 2 weeks. One month ago, he received care in the emergency\
+  \ department for pain over his right flank following a motor vehicle collision.\
+  \ At that time, blood pressure was 160/100 mm Hg and CT scan of the abdomen showed\
+  \ an incidental 10-cm left adrenal mass. Results of laboratory studies, including\
+  \ complete blood count, serum electrolyte concentrations, and liver function tests,\
+  \ were within the reference ranges. The patient otherwise had been healthy and had\
+  \ never been told that he had elevated blood pressure. He takes no medications.\
+  \ A follow-up visit in the office 2 weeks ago disclosed elevated urinary normetanephrine\
+  \ and metanephrine and plasma aldosterone concentrations. The patient was referred\
+  \ to a surgeon, who recommended the adrenalectomy. Today, vital signs are temperature\
+  \ 36.6\xB0C (97.9\xB0F), pulse 100/min, respirations 14/min, and blood pressure\
+  \ 170/95 mm Hg. Physical examination discloses no significant findings. Initial\
+  \ preoperative preparation should include treatment with which of the following?\n\
+  (A) Labetalol (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxybenzamine\n\
+  A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
+  \ The symptoms and the adrenal mass suggested pheochromocytoma, and the blood pressure\
+  \ indicates hypertension. Phenoxybenzamine is used to treat hypertension caused\
+  \ by pheochromocytoma. The answer is (D)."
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_professional_medicine

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml

+dataset_name: professional_psychology
+description: "The following are multiple choice questions (with answers) about professional\
+  \ psychology.\n\nQ: In the construction of a multiple regression equation for purposes\
+  \ of prediction, the optimal combination of measures is one in which the predictors\n\
+  (A) are uncorrelated with each other but are moderately correlated with the criterion\
+  \ (B) have low correlations with each other and low correlations with the criterion\
+  \ (C) are highly intercorrelated with each other and moderately correlated with\
+  \ the criterion (D) have low correlations with the criterion bur are moderately\
+  \ correlated with each other\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on psychology for help. The basis of multiple regression is to assess\
+  \ the relationship between one continuous variable and a set of independent variables.\
+  \ So the predictors should be uncorrelated with each other but are moderately correlated\
+  \ with the criterion. The answer is (A).\n\nQ: There are three ways to measure the\
+  \ Central Tendency: the Mean, the Median and the Mode. From your knowledge about\
+  \ them, what is the mode?\n(A) less sensitive to extreme scores than the mean (B)\
+  \ more useful for skewed distributions (C) sensitive to extreme values and highly\
+  \ skewed distributions (D) the most frequently occurring number\nA: Let's think\
+  \ step by step. We refer to Wikipedia articles on psychology for help. The definition\
+  \ of mode is the most frequently occurring number. The answer is (D).\n\nQ: Carl\
+  \ Jung believed that a client's transference:\n(A) is a fantasy that distracts the\
+  \ client from reality. (B) represents \u201Cmixed feelings\u201D toward the therapist.\
+  \ (C) \"is a form of \"\"acting out.\"\"\" (D) reflects the client\u2019s personal\
+  \ and collective unconscious.\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on psychology for help. Transference is a phenomenon that a person's\
+  \ feelings are unconsciously redirected, so it reflects the client's personal and\
+  \ collective unconscious. The answer is (D).\n\nQ: In terms of Hofstede\u2019s (1980)\
+  \ five cultural dimensions, the United States scores at the top of the scale on:\n\
+  (A) individualism. (B) individualism and power distance. (C) power distance and\
+  \ masculinity. (D) uncertainty avoidance.\nA: Let's think step by step. We refer\
+  \ to Wikipedia articles on psychology for help. US scores highest on individualism\
+  \ among the five cultural dimensions. The answer is (A).\n\nQ: One of your therapy\
+  \ clients asks your advice about a good weight- reduction program. You have investigated\
+  \ the programs in the community and are enrolled in the one you consider the best.\
+  \ This program offers a $50 bonus to its patrons for each new person they bring\
+  \ into the program. Under these circumstances, your most appropriate response would\
+  \ be to\n(A) tell your client the pros and cons of each program you know about except\
+  \ for the one in which you are enrolled (B) recommend to your client the program\
+  \ in which you are enrolled and explain the $50 bonus you will receive (C) recommend\
+  \ to your client the program in which you are enrolled and offer to have the $50\
+  \ bonus credited to your client's account in the program (D) tell your client the\
+  \ pros and cons of each program you know about, but do not claim the $50 bonus if\
+  \ your client enrolls in your program\nA: Let's think step by step. We refer to\
+  \ Wikipedia articles on psychology for help. Based on the circumstances, you should\
+  \ tell your client about the pros and cons of each program, but it would be inappropriate\
+  \ to receive the bonus, so you should not claim the $50 bonus. The answer is (D)."
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_professional_psychology

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml

+dataset_name: public_relations
+description: 'The following are multiple choice questions (with answers) about public
+  relations.
+
+
+  Q: Earth Hour was a campaign launched by which organization?
+
+  (A) Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund
+
+  A: Let''s think step by step. We refer to Wikipedia articles on public relations
+  for help. Earth Hour is a worldwide movement oragnized launched by the World Wildlife
+  Fund. The answer is (D).
+
+
+  Q: In issues management, what is the most proactive approach to addressing negative
+  or misleading information posted online about your organization?
+
+  (A) Buy domain names that could be used by opposition groups. (B) Post anonymous
+  comments on blogs to combat this information. (C) Prepare a news release that discredits
+  the inaccurate information. (D) Make policy changes to address complaints highlighted
+  on these sites.
+
+  A: Let''s think step by step. We refer to Wikipedia articles on public relations
+  for help. In issues management, the most proactive approach to addressing negative
+  or misleading information posted online is to make policy changes to address complaints
+  highlighted on those sites. The answer is (D).
+
+
+  Q: At which stage in the planning process would a situation analysis be carried
+  out?
+
+  (A) Defining the program (B) Planning the program (C) Taking action and implementing
+  ideas (D) Evaluation of the program
+
+  A: Let''s think step by step. We refer to Wikipedia articles on public relations
+  for help. Situation analyses are typically carried out during the planning process
+  stage of defining the program. The answer is (A).
+
+
+  Q: Which of these statements is true of the Vatican in 2010 at the time of the accusations
+  of child abuse cover-ups?
+
+  (A) There was a coordinated media response. (B) Consistent messages were communicated.
+  (C) Criticisms were taken as attacks on the Catholic Church. (D) The credibility
+  of the Vatican was upheld.
+
+  A: Let''s think step by step. We refer to Wikipedia articles on public relations
+  for help. In 2010 when there were accusations of child abuse cover-ups, the Vatican
+  took those criticisms as attacks on the Catholic Church. The answer is (C).
+
+
+  Q: What should a public relations media practitioner do if she does not know the
+  answer to a reporter''s question?
+
+  (A) Give the reporter other information she is certain is correct. (B) Say that
+  the information is ''off the record'' and will be disseminated later. (C) Say ''I
+  don''t know'' and promise to provide the information later. (D) Say ''no comment,''
+  rather than appear uninformed.
+
+  A: Let''s think step by step. We refer to Wikipedia articles on public relations
+  for help. If a public relations media practitioner does not know the answer to a
+  reporter''s question, they should say ''I don''t know'' and offer to provide the
+  information later. The answer is (C).'
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_public_relations

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml

+dataset_name: security_studies
+description: "The following are multiple choice questions (with answers) about security\
+  \ studies.\n\nQ: What are the frameworks of analysis within which terrorism has\
+  \ been considered (as of 2020)?\n(A) Competition between larger nations has resulted\
+  \ in some countries actively supporting terrorist groups to undermine the strength\
+  \ of rival states. Terrorist networks are extended patronage clubs maintained and\
+  \ paid for by their donor states and are conceptualised as being like state actors,\
+  \ to be dealt with using military force. (B) Globalization has enabled the internationalization\
+  \ of terrorist activities by opening up their operational space, although coordination\
+  \ is still managed from a geographical base. This suggests that terrorist groups\
+  \ are nationally structured which means that terrorism cannot be considered in terms\
+  \ of a war to be defeated militarily without having serious implications on the\
+  \ indigenous population. (C) Terrorism can be viewed as a problem to be resolved\
+  \ by military means (war on terrorism), by normal police techniques (terrorism as\
+  \ crime), or as a medical problem with underlying causes and symptoms (terrorism\
+  \ as disease). (D) Terrorism is viewed as a criminal problem. The criminalization\
+  \ of terrorism has two important implications. Firstly, it suggests that terrorism\
+  \ can be eradicated - terrorists can be caught and brought to trial by normal judicial\
+  \ proceedings thereby removing the threat from society - and secondly, it suggests\
+  \ that preventative crime techniques are applicable to prevent its development.\n\
+  A: Let's think step by step. We refer to Wikipedia articles on security studies\
+  \ for help. (A) is wrong because it is not competition between larger nations that\
+  \ causes terrorism. \n(B) is wrong because globalization is not the cause of terrorism.\n\
+  (C) is correct because the US undertook the war on terrorism. \n(D) is wrong because\
+  \ preventative crime techniques will likely not end terrorism. The answer is (C).\n\
+  \nQ: Which of the following is the best lens through which to investigate the role\
+  \ of child soldiers?\n(A) Child soldiers are victims of combat that need re-education\
+  \ and rehabilitation. (B) Children and their mothers are not active subjects in\
+  \ warfare and are best considered as subjects in the private sphere. (C) Children\
+  \ are most often innocent bystanders in war and are best used as signifiers of peace.\
+  \ (D) Children have political subjecthood that is missed when they are considered\
+  \ as passive victims of warfare.\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on security studies for help. Child soliders as a political topic can\
+  \ be missed when they are considered passive victims of warfare. The answer is (D).\n\
+  \nQ: How can we best describe the relationship between the state-centric approach\
+  \ and the concept of human security?\n(A) There are such wide divisions within the\
+  \ human security framework regarding the nature of threats and referent objects\
+  \ that no widely applicable comparisons between state-centric approaches and human\
+  \ security can be drawn. (B) By adopting the framework of human security, the limitations\
+  \ of the realist state-centric approach become evident. Whilst human security defines\
+  \ the referent object as the person or population, state-centric approaches prioritise\
+  \ the security of the state, de-prioritizing the pursuit of human security. (C)\
+  \ The state-centric approach to security is a faction of human security, usually\
+  \ defined within the broad school of human security. By being state-centric this\
+  \ approach prioritises the individual as the referent object in security studies.\
+  \ (D) Both the state-centric and human-centric approaches to security are mutually\
+  \ exclusive and offer a sufficient analytic framework with which to understand the\
+  \ international security system. It is therefore the role of security analysts to\
+  \ determine which of these substantial concepts is correct, and which should be\
+  \ discarded.\nA: Let's think step by step. We refer to Wikipedia articles on security\
+  \ studies for help. Human security focuses on a person or population whereas state-centric\
+  \ approaches focus on the state while deprioritizing human security. The answer\
+  \ is (B).\n\nQ: In order to become securitized, a threat must be presented in which\
+  \ of these ways?\n(A) As an existential threat that requires immediate and extraordinary\
+  \ action, posing a threat to the survival of the state or to societal security.\
+  \ (B) As requiring immediate and extraordinary action by the state, threatening\
+  \ the survival of a referent object and therefore warranting the use of measures\
+  \ not normally employed in the political realm. (C) As an urgent threat to the survival\
+  \ of the referent object, so serious that it legitimises the employment of extraordinary\
+  \ action in response. (D) As an urgent threat to the survival of the audience that\
+  \ requires extraordinary or emergency measures.\nA: Let's think step by step. We\
+  \ refer to Wikipedia articles on security studies for help. To be securitized, a\
+  \ threat must be an urgent threat to the survival of the referent object. The answer\
+  \ is (C).\n\nQ: What distinguishes coercive diplomacy from military force?\n(A)\
+  \ Compellence is another term for coercive diplomacy, but covering a narrower set\
+  \ of criteria; compellence covers those threats aimed at initiating adversary action.\
+  \ A threat to coerce a state to give up part of its territory would count as coercive\
+  \ diplomacy, as long as that threat proactively initiates action before reactive\
+  \ diplomacy is taken. (B) Coercive diplomacy constitutes the threats of limited\
+  \ force to induce adversary's incentive to comply with the coercer's demands. It\
+  \ is an influence strategy that is intended to obtain compliance: the use of force\
+  \ to defeat an opponent first does not count. It leaves an element of choice with\
+  \ the target to comply, or to continue. (C) Military force, or the threat of military\
+  \ force, utilises fear to achieve strategic objectives. Coercive diplomacy is differentiated\
+  \ from this approach, because it does not use fear as a tool for coercing an adversary.\
+  \ (D) Coercive diplomacy is employed to use force but to limit its effects on the\
+  \ international community. Coercive diplomacy is an aggressive strategy that is\
+  \ intended to obtain compliance through defeat. It does not leave an element of\
+  \ choice with the target, the target either being forced to comply or engage in\
+  \ conflict. It seeks to control by imposing compliance by removing any opportunity\
+  \ for negotiation or concession.\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on security studies for help. Coercive diplomacy uses the threat of force\
+  \ to induce the opponent to comply with demands. The answer is (B)."
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_security_studies

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml

+dataset_name: sociology
+description: 'The following are multiple choice questions (with answers) about sociology.
+
+
+  Q: Which of the following is not a problem associated with official statistics on
+  strike action?
+
+  (A) most strikes go unnoticed by employers and the mass media (B) not all industrial
+  disputes will be reported by the employer (C) the definition of strikes excludes
+  those that involve fewer than ten workers or last less than one day (D) it is hard
+  to compare strikes that were measured in different ways
+
+  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
+  Official statistics on strike action can be problematic because not all industrial
+  disputes will be reported by employers, the definition of strikes excludes those
+  that involves fewer than ten workers or last less than one day, and it is hard to
+  compare strikes that were measured in different ways. Thus, (A) is not a problem
+  associated with official statistics on strike action. The answer is (A).
+
+
+  Q: What does Berger (1963) describe as a metaphor for social reality?
+
+  (A) a fairground ride (B) a circus (C) a puppet theatre (D) a ballet
+
+  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
+  Berger describes social reality using the metaphor of a puppet theatre. The answer
+  is (C).
+
+
+  Q: The term ''hegemony'' refers to:
+
+  (A) the tendency for the working class not to realize their own interests (B) a
+  dominant ideology that legitimates economic, political and cultural power (C) a
+  form of dual consciousness based on ideology and everyday experiences (D) a mode
+  of payment given for outstanding topiary
+
+  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
+  Hegemony refers to a dominant ideology that legitimates economic, policital, and
+  cultural power. The answer is (B).
+
+
+  Q: The shift from ''civil religion'' to ''common religion'' means that:
+
+  (A) the increasing bureaucracy of the state has made religion only a marginal part
+  of our lives (B) despite the weakening of traditional authority, our everyday lives
+  and ''common sense'' remain shaped by religious beliefs and values (C) religious
+  participation in collective worship may have declined, but people still practise
+  their faiths in private (D) people are much more likely to discuss their religious
+  beliefs in public, informal settings
+
+  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
+  The shift from civil religion to common religion means that despite the weakening
+  of traditional authority, our everyday lives and common sense remain shaped by religious
+  beliefs and values. The answer is (B).
+
+
+  Q: Which of the following did the post-war welfare state of 1948 not aim to provide:
+
+  (A) free health care and education for all (B) a minimum wage (C) full employment
+  (D) universal welfare
+
+  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
+  The post-war welfare state of 1948 aimed to provide free healthcare and education,
+  full employment, and universal welfare. But it did not aim to provide a minimum
+  wage. The answer is (B).'
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_sociology

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml

+dataset_name: us_foreign_policy
+description: 'The following are multiple choice questions (with answers) about us
+  foreign policy.
+
+
+  Q: How did Donald Trump attack globalization in the 2016 campaign?
+
+  (A) Globalization had made men like him too rich (B) Globalization only benefited
+  certain American states, such as New York (C) Liberal elites had encouraged globalization,
+  while ''ordinary Americans'' lost jobs because of it (D) Globalization encouraged
+  damaging trade wars
+
+  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
+  for help. Trump attacked globalization because he believed ordinary Americans lost
+  jobs due to it, and so he wanted to blame liberals who had encouraged it. The answer
+  is (C).
+
+
+  Q: How did NSC-68 change U.S. strategy?
+
+  (A) It globalized containment. (B) It militarized containment. (C) It called for
+  the development of the hydrogen bomb. (D) All of the above
+
+  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
+  for help. NSC-68 outlined a variety of courses of action, including globalization
+  of containment, militarization of contaiment, and the development of the hydrogen
+  bomb. The answer is (D).
+
+
+  Q: How do Defensive Realism and Offensive Realism differ in their explanation of
+  state behaviour?
+
+  (A) Defensive realists place greater emphasis on the role of international institutions
+  (B) Defensive realists place less emphasis on geographical factors (C) Offensive
+  realists give more priority to the national interest than Defensive realists. (D)
+  Defensive realists believe states are security maximizers, while Offensive realists
+  believe states to be power maximizers
+
+  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
+  for help. While defensive realism advocates that states are security maximizers,
+  offensive realists think of states as power maximizers. The answer is (D).
+
+
+  Q: The realm of policy decisions concerned primarily with relations between the
+  United States and the rest of the world is known as
+
+  (A) terrorism policy. (B) economic policy. (C) foreign policy. (D) international
+  policy.
+
+  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
+  for help. The topic of policy decisions concerns with relations between the US and
+  the rest of the world is known as foreign policy. The answer is (C).
+
+
+  Q: How did the 2008 financial crisis affect America''s international reputation?
+
+  (A) It damaged support for the US model of political economy and capitalism (B)
+  It created anger at the United States for exaggerating the crisis (C) It increased
+  support for American global leadership under President Obama (D) It reduced global
+  use of the US dollar
+
+  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
+  for help. The 2008 financial crisis damanged the international reputation of the
+  American model of political economy and capitalism. The answer is (A).'
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_us_foreign_policy

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml

+dataset_name: virology
+description: 'The following are multiple choice questions (with answers) about virology.
+
+
+  Q: The median survival time to AIDS and death was established by following:
+
+  (A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters
+  (D) High-risk seronegatives
+
+  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
+  The median survival time to AIDS and death was established as a result of the development
+  of seroconverters. The answer is (C).
+
+
+  Q: Which of the following is a morphological characteristic of the paramyxoviruses.
+
+  (A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate
+  viruses (C) Icosahedral viruses with envelope (D) Very large viruses
+
+  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
+  Paramyxoviruses are fragile viruses often visualised with RNA spewing from the inside.
+  The answer is (A).
+
+
+  Q: The most important goal of a behavioral intervention is:
+
+  (A) Change in behavior (B) Comprehensive coverage (C) Effective use of behavioral
+  theory (D) Sustained behavior change
+
+  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
+  The prim goal of a behavioral intervention is to cause sustained behavior change.
+  The answer is (D).
+
+
+  Q: A key factor facilitating the application of nested case-control studies from
+  the MACS was:
+
+  (A) Data collection (B) Establishment of a repository of biologic specimens (C)
+  Participant interest (D) Administration of the questionnaire by staff
+
+  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
+  The Multicenter AIDS Cohort Study''s use of nested case-control studies was facilitated
+  by the establishment of a repository of biologic specimens. The answer is (B).
+
+
+  Q: Why are parvoviruses a highly impactful parasite?
+
+  (A) Because they have no nucleic acid (B) They require a helper virus (C) Only replicate
+  in dividing cells (D) Can integrate into host chromosomes
+
+  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
+  Paroviruses are highly impactful because they do not have nucleic acid. The answer
+  is (A).'
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_virology

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml

+dataset_name: world_religions
+description: 'The following are multiple choice questions (with answers) about world
+  religions.
+
+
+  Q: How can the Upanishads be characterized?
+
+  (A) Ritual texts (B) Philosophical texts (C) Hymns (D) Origin stories
+
+  A: Let''s think step by step. We refer to Wikipedia articles on world religions
+  for help. The Upanishads are the most recent part of Vedas (the oldest scriptures
+  in Hinduism) and supplied the basis of later Hindu philosophy. So they are philosophical
+  texts. The answer is (B).
+
+
+  Q: What is the Second Gem in Buddhism?
+
+  (A) The Dharma (B) The Sangha (C) The Buddha (D) The Bodhisattva
+
+  A: Let''s think step by step. We refer to Wikipedia articles on world religions
+  for help. The Second Gem in Buddhism is The Dharma. The answer is (A).
+
+
+  Q: Which Japanese government promoted a kind of national cult based on the emperor
+  and his associations with kami?
+
+  (A) Honen (B) Tanaka (C) Tokugawa (D) Meiji
+
+  A: Let''s think step by step. We refer to Wikipedia articles on world religions
+  for help. The promotion of a national cult based on the emperor and his associations
+  with Kami happened during the reign of Emperor Meiji (1852-1912). The answer is
+  (D).
+
+
+  Q: In which dynasty was the "Mandate of Heaven" developed to legitimatize the new
+  rulers?
+
+  (A) Shang (B) Zhou (C) Han (D) Xia
+
+  A: Let''s think step by step. We refer to Wikipedia articles on world religions
+  for help. The "Mandate of Heaven" was developed as an ancient Chinese philosophical
+  concept during the Zhou Dynasty (1046-256 BCE). The answer is (B).
+
+
+  Q: What is the sign of the covenant for Jewish males?
+
+  (A) The rainbow (B) Circumcision (C) A son (D) Bar mitzvah
+
+  A: Let''s think step by step. We refer to Wikipedia articles on world religions
+  for help. In Judaism, the most distinctive sign of the covenant is circumcision
+  (brit milah). The answer is (B).'
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_world_religions

lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_generative_template_yaml

+group: mmlu_flan_cot_zeroshot
+dataset_path: cais/mmlu
+validation_split: validation
+fewshot_split: dev
+output_type: greedy_until
+doc_to_text: "Q: {{question.strip()}}\n(A) {{choices[0]}} (B) {{choices[1]}} (C) {{choices[2]}} (D) {{choices[3]}}\nA: Let's think step by step."
+doc_to_target: "{{['(A)', '(B)', '(C)', '(D)'][answer]}}"
+filter_list:
+  - name: "get-answer"
+    filter:
+      - function: "regex"
+        regex_pattern: "((?<=The answer is )(.*)(?=.)|(?<=the answer is )(.*)(?=.)|(?<=The answer: )(.*)(?=.)|(?<=The final answer: )(.*)(?=.))"
+      - function: "take_first"
+generation_kwargs:
+  until:
+    - "</s>"
+  do_sample: false
+  temperature: 0.0
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml

+dataset_name: abstract_algebra
+description: 'The following are multiple choice questions (with answers) about abstract
+  algebra.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_abstract_algebra

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml

+dataset_name: anatomy
+description: 'The following are multiple choice questions (with answers) about anatomy.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_anatomy

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml

+dataset_name: astronomy
+description: 'The following are multiple choice questions (with answers) about astronomy.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_astronomy

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml

+dataset_name: business_ethics
+description: 'The following are multiple choice questions (with answers) about business
+  ethics.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_business_ethics

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml

+dataset_name: clinical_knowledge
+description: 'The following are multiple choice questions (with answers) about clinical
+  knowledge.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_clinical_knowledge

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml

+dataset_name: college_biology
+description: 'The following are multiple choice questions (with answers) about college
+  biology.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_college_biology

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml

+dataset_name: college_chemistry
+description: 'The following are multiple choice questions (with answers) about college
+  chemistry.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_college_chemistry

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml

+dataset_name: college_computer_science
+description: 'The following are multiple choice questions (with answers) about college
+  computer science.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_college_computer_science

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml

+dataset_name: college_mathematics
+description: 'The following are multiple choice questions (with answers) about college
+  mathematics.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_college_mathematics

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml

+dataset_name: college_medicine
+description: 'The following are multiple choice questions (with answers) about college
+  medicine.
+
+
+  '
+include: _mmlu_flan_generative_template_yaml
+task: mmlu_flan_cot_zeroshot_college_medicine