Merge branch 'big-refactor' into verbosity-rework

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml

-dataset_name: professional_medicine
-description: "The following are multiple choice questions (with answers) about professional\
+"dataset_name": "professional_medicine"
+"description": "The following are multiple choice questions (with answers) about professional\
   \ medicine.\n\nQ: A 22-year-old male marathon runner presents to the office with\
   \ the complaint of right-sided rib pain when he runs long distances. Physical examination\
-  \ reveals normal heart and lung findings and an exhalation dysfunction at ribs\_\
-  4-5 on the right. Which of the following muscles or muscle groups will be most useful\
+  \ reveals normal heart and lung findings and an exhalation dysfunction at ribs 4-5\
+  \ on the right. Which of the following muscles or muscle groups will be most useful\
   \ in correcting this dysfunction utilizing a direct method?\n(A) anterior scalene\
   \ (B) latissimus dorsi (C) pectoralis minor (D) quadratus lumborum\nA: Let's think\
   \ step by step. We refer to Wikipedia articles on medicine for help. Among the options,\
   \ only pectoralis minor muscle origins from the outer surfaces of the 3rd to 5th\
   \ ribs. The answer is (C).\n\nQ: A 36-year-old male presents to the office with\
-  \ a\_3-week\_history of low back pain. He denies any recent trauma but says that\
-  \ he climbs in and out of his truck numerous times a day for his job. Examination\
-  \ of the patient in the prone position reveals a deep sacral sulcus on the left,\
-  \ a posterior inferior lateral angle on the right, and a lumbosacral junction that\
+  \ a 3-week history of low back pain. He denies any recent trauma but says that he\
+  \ climbs in and out of his truck numerous times a day for his job. Examination of\
+  \ the patient in the prone position reveals a deep sacral sulcus on the left, a\
+  \ posterior inferior lateral angle on the right, and a lumbosacral junction that\
   \ springs freely on compression. The most likely diagnosis is\n(A) left-on-left\
   \ sacral torsion (B) left-on-right sacral torsion (C) right unilateral sacral flexion\
   \ (D) right-on-right sacral torsion\nA: Let's think step by step. We refer to Wikipedia\
@@ -23,9 +23,9 @@ description: "The following are multiple choice questions (with answers) about p
   \ nonproductive cough, runny nose, and frontal headache. He says the headache is\
   \ worse in the morning and ibuprofen does provide some relief. He has not had shortness\
   \ of breath. Medical history is unremarkable. He takes no medications other than\
-  \ the ibuprofen for pain. Vital signs are temperature 37.4\xB0C (99.4\xB0F), pulse\
-  \ 88/min, respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the\
-  \ nares shows erythematous mucous membranes. Examination of the throat shows erythema\
+  \ the ibuprofen for pain. Vital signs are temperature 37.4°C (99.4°F), pulse 88/min,\
+  \ respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the nares\
+  \ shows erythematous mucous membranes. Examination of the throat shows erythema\
   \ and follicular lymphoid hyperplasia on the posterior oropharynx. There is no palpable\
   \ cervical adenopathy. Lungs are clear to auscultation. Which of the following is\
   \ the most likely cause of this patient's symptoms?\n(A) Allergic rhinitis (B) Epstein-Barr\
@@ -57,13 +57,14 @@ description: "The following are multiple choice questions (with answers) about p
   \ A follow-up visit in the office 2 weeks ago disclosed elevated urinary normetanephrine\
   \ and metanephrine and plasma aldosterone concentrations. The patient was referred\
   \ to a surgeon, who recommended the adrenalectomy. Today, vital signs are temperature\
-  \ 36.6\xB0C (97.9\xB0F), pulse 100/min, respirations 14/min, and blood pressure\
-  \ 170/95 mm Hg. Physical examination discloses no significant findings. Initial\
-  \ preoperative preparation should include treatment with which of the following?\n\
-  (A) Labetalol (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxybenzamine\n\
+  \ 36.6°C (97.9°F), pulse 100/min, respirations 14/min, and blood pressure 170/95\
+  \ mm Hg. Physical examination discloses no significant findings. Initial preoperative\
+  \ preparation should include treatment with which of the following?\n(A) Labetalol\
+  \ (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxybenzamine\n\
   A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
   \ The symptoms and the adrenal mass suggested pheochromocytoma, and the blood pressure\
   \ indicates hypertension. Phenoxybenzamine is used to treat hypertension caused\
   \ by pheochromocytoma. The answer is (D)."
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_professional_medicine
+"group": "mmlu_flan_cot_fewshot_other"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_professional_medicine"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml

-dataset_name: professional_psychology
-description: "The following are multiple choice questions (with answers) about professional\
+"dataset_name": "professional_psychology"
+"description": "The following are multiple choice questions (with answers) about professional\
   \ psychology.\n\nQ: In the construction of a multiple regression equation for purposes\
   \ of prediction, the optimal combination of measures is one in which the predictors\n\
   (A) are uncorrelated with each other but are moderately correlated with the criterion\
@@ -18,30 +18,31 @@ description: "The following are multiple choice questions (with answers) about p
   \ step by step. We refer to Wikipedia articles on psychology for help. The definition\
   \ of mode is the most frequently occurring number. The answer is (D).\n\nQ: Carl\
   \ Jung believed that a client's transference:\n(A) is a fantasy that distracts the\
-  \ client from reality. (B) represents \u201Cmixed feelings\u201D toward the therapist.\
-  \ (C) \"is a form of \"\"acting out.\"\"\" (D) reflects the client\u2019s personal\
-  \ and collective unconscious.\nA: Let's think step by step. We refer to Wikipedia\
-  \ articles on psychology for help. Transference is a phenomenon that a person's\
-  \ feelings are unconsciously redirected, so it reflects the client's personal and\
-  \ collective unconscious. The answer is (D).\n\nQ: In terms of Hofstede\u2019s (1980)\
-  \ five cultural dimensions, the United States scores at the top of the scale on:\n\
-  (A) individualism. (B) individualism and power distance. (C) power distance and\
-  \ masculinity. (D) uncertainty avoidance.\nA: Let's think step by step. We refer\
-  \ to Wikipedia articles on psychology for help. US scores highest on individualism\
-  \ among the five cultural dimensions. The answer is (A).\n\nQ: One of your therapy\
-  \ clients asks your advice about a good weight- reduction program. You have investigated\
-  \ the programs in the community and are enrolled in the one you consider the best.\
-  \ This program offers a $50 bonus to its patrons for each new person they bring\
-  \ into the program. Under these circumstances, your most appropriate response would\
-  \ be to\n(A) tell your client the pros and cons of each program you know about except\
-  \ for the one in which you are enrolled (B) recommend to your client the program\
-  \ in which you are enrolled and explain the $50 bonus you will receive (C) recommend\
-  \ to your client the program in which you are enrolled and offer to have the $50\
-  \ bonus credited to your client's account in the program (D) tell your client the\
-  \ pros and cons of each program you know about, but do not claim the $50 bonus if\
-  \ your client enrolls in your program\nA: Let's think step by step. We refer to\
-  \ Wikipedia articles on psychology for help. Based on the circumstances, you should\
-  \ tell your client about the pros and cons of each program, but it would be inappropriate\
-  \ to receive the bonus, so you should not claim the $50 bonus. The answer is (D)."
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_professional_psychology
+  \ client from reality. (B) represents “mixed feelings” toward the therapist. (C)\
+  \ \"is a form of \"\"acting out.\"\"\" (D) reflects the client’s personal and collective\
+  \ unconscious.\nA: Let's think step by step. We refer to Wikipedia articles on psychology\
+  \ for help. Transference is a phenomenon that a person's feelings are unconsciously\
+  \ redirected, so it reflects the client's personal and collective unconscious. The\
+  \ answer is (D).\n\nQ: In terms of Hofstede’s (1980) five cultural dimensions, the\
+  \ United States scores at the top of the scale on:\n(A) individualism. (B) individualism\
+  \ and power distance. (C) power distance and masculinity. (D) uncertainty avoidance.\n\
+  A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
+  \ US scores highest on individualism among the five cultural dimensions. The answer\
+  \ is (A).\n\nQ: One of your therapy clients asks your advice about a good weight-\
+  \ reduction program. You have investigated the programs in the community and are\
+  \ enrolled in the one you consider the best. This program offers a $50 bonus to\
+  \ its patrons for each new person they bring into the program. Under these circumstances,\
+  \ your most appropriate response would be to\n(A) tell your client the pros and\
+  \ cons of each program you know about except for the one in which you are enrolled\
+  \ (B) recommend to your client the program in which you are enrolled and explain\
+  \ the $50 bonus you will receive (C) recommend to your client the program in which\
+  \ you are enrolled and offer to have the $50 bonus credited to your client's account\
+  \ in the program (D) tell your client the pros and cons of each program you know\
+  \ about, but do not claim the $50 bonus if your client enrolls in your program\n\
+  A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
+  \ Based on the circumstances, you should tell your client about the pros and cons\
+  \ of each program, but it would be inappropriate to receive the bonus, so you should\
+  \ not claim the $50 bonus. The answer is (D)."
+"group": "mmlu_flan_cot_fewshot_social_sciences"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_professional_psychology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml

-dataset_name: public_relations
-description: 'The following are multiple choice questions (with answers) about public
-  relations.
-
-
-  Q: Earth Hour was a campaign launched by which organization?
-
-  (A) Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund
-
-  A: Let''s think step by step. We refer to Wikipedia articles on public relations
-  for help. Earth Hour is a worldwide movement oragnized launched by the World Wildlife
-  Fund. The answer is (D).
-
-
-  Q: In issues management, what is the most proactive approach to addressing negative
-  or misleading information posted online about your organization?
-
-  (A) Buy domain names that could be used by opposition groups. (B) Post anonymous
-  comments on blogs to combat this information. (C) Prepare a news release that discredits
-  the inaccurate information. (D) Make policy changes to address complaints highlighted
-  on these sites.
-
-  A: Let''s think step by step. We refer to Wikipedia articles on public relations
-  for help. In issues management, the most proactive approach to addressing negative
-  or misleading information posted online is to make policy changes to address complaints
-  highlighted on those sites. The answer is (D).
-
-
-  Q: At which stage in the planning process would a situation analysis be carried
-  out?
-
-  (A) Defining the program (B) Planning the program (C) Taking action and implementing
-  ideas (D) Evaluation of the program
-
-  A: Let''s think step by step. We refer to Wikipedia articles on public relations
-  for help. Situation analyses are typically carried out during the planning process
-  stage of defining the program. The answer is (A).
-
-
-  Q: Which of these statements is true of the Vatican in 2010 at the time of the accusations
-  of child abuse cover-ups?
-
-  (A) There was a coordinated media response. (B) Consistent messages were communicated.
-  (C) Criticisms were taken as attacks on the Catholic Church. (D) The credibility
-  of the Vatican was upheld.
-
-  A: Let''s think step by step. We refer to Wikipedia articles on public relations
-  for help. In 2010 when there were accusations of child abuse cover-ups, the Vatican
-  took those criticisms as attacks on the Catholic Church. The answer is (C).
-
-
-  Q: What should a public relations media practitioner do if she does not know the
-  answer to a reporter''s question?
-
-  (A) Give the reporter other information she is certain is correct. (B) Say that
-  the information is ''off the record'' and will be disseminated later. (C) Say ''I
-  don''t know'' and promise to provide the information later. (D) Say ''no comment,''
-  rather than appear uninformed.
-
-  A: Let''s think step by step. We refer to Wikipedia articles on public relations
-  for help. If a public relations media practitioner does not know the answer to a
-  reporter''s question, they should say ''I don''t know'' and offer to provide the
-  information later. The answer is (C).'
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_public_relations
+"dataset_name": "public_relations"
+"description": "The following are multiple choice questions (with answers) about public\
+  \ relations.\n\nQ: Earth Hour was a campaign launched by which organization?\n(A)\
+  \ Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund\nA: Let's think step by\
+  \ step. We refer to Wikipedia articles on public relations for help. Earth Hour\
+  \ is a worldwide movement oragnized launched by the World Wildlife Fund. The answer\
+  \ is (D).\n\nQ: In issues management, what is the most proactive approach to addressing\
+  \ negative or misleading information posted online about your organization?\n(A)\
+  \ Buy domain names that could be used by opposition groups. (B) Post anonymous comments\
+  \ on blogs to combat this information. (C) Prepare a news release that discredits\
+  \ the inaccurate information. (D) Make policy changes to address complaints highlighted\
+  \ on these sites.\nA: Let's think step by step. We refer to Wikipedia articles on\
+  \ public relations for help. In issues management, the most proactive approach to\
+  \ addressing negative or misleading information posted online is to make policy\
+  \ changes to address complaints highlighted on those sites. The answer is (D).\n\
+  \nQ: At which stage in the planning process would a situation analysis be carried\
+  \ out?\n(A) Defining the program (B) Planning the program (C) Taking action and\
+  \ implementing ideas (D) Evaluation of the program\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on public relations for help. Situation analyses\
+  \ are typically carried out during the planning process stage of defining the program.\
+  \ The answer is (A).\n\nQ: Which of these statements is true of the Vatican in 2010\
+  \ at the time of the accusations of child abuse cover-ups?\n(A) There was a coordinated\
+  \ media response. (B) Consistent messages were communicated. (C) Criticisms were\
+  \ taken as attacks on the Catholic Church. (D) The credibility of the Vatican was\
+  \ upheld.\nA: Let's think step by step. We refer to Wikipedia articles on public\
+  \ relations for help. In 2010 when there were accusations of child abuse cover-ups,\
+  \ the Vatican took those criticisms as attacks on the Catholic Church. The answer\
+  \ is (C).\n\nQ: What should a public relations media practitioner do if she does\
+  \ not know the answer to a reporter's question?\n(A) Give the reporter other information\
+  \ she is certain is correct. (B) Say that the information is 'off the record' and\
+  \ will be disseminated later. (C) Say 'I don't know' and promise to provide the\
+  \ information later. (D) Say 'no comment,' rather than appear uninformed.\nA: Let's\
+  \ think step by step. We refer to Wikipedia articles on public relations for help.\
+  \ If a public relations media practitioner does not know the answer to a reporter's\
+  \ question, they should say 'I don't know' and offer to provide the information\
+  \ later. The answer is (C)."
+"group": "mmlu_flan_cot_fewshot_social_sciences"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_public_relations"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml

-dataset_name: security_studies
-description: "The following are multiple choice questions (with answers) about security\
+"dataset_name": "security_studies"
+"description": "The following are multiple choice questions (with answers) about security\
   \ studies.\n\nQ: What are the frameworks of analysis within which terrorism has\
   \ been considered (as of 2020)?\n(A) Competition between larger nations has resulted\
   \ in some countries actively supporting terrorist groups to undermine the strength\
@@ -81,5 +81,6 @@ description: "The following are multiple choice questions (with answers) about s
   \ for negotiation or concession.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on security studies for help. Coercive diplomacy uses the threat of force\
   \ to induce the opponent to comply with demands. The answer is (B)."
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_security_studies
+"group": "mmlu_flan_cot_fewshot_social_sciences"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_security_studies"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml

-dataset_name: sociology
-description: 'The following are multiple choice questions (with answers) about sociology.
-
-
-  Q: Which of the following is not a problem associated with official statistics on
-  strike action?
-
-  (A) most strikes go unnoticed by employers and the mass media (B) not all industrial
-  disputes will be reported by the employer (C) the definition of strikes excludes
-  those that involve fewer than ten workers or last less than one day (D) it is hard
-  to compare strikes that were measured in different ways
-
-  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
-  Official statistics on strike action can be problematic because not all industrial
-  disputes will be reported by employers, the definition of strikes excludes those
-  that involves fewer than ten workers or last less than one day, and it is hard to
-  compare strikes that were measured in different ways. Thus, (A) is not a problem
-  associated with official statistics on strike action. The answer is (A).
-
-
-  Q: What does Berger (1963) describe as a metaphor for social reality?
-
-  (A) a fairground ride (B) a circus (C) a puppet theatre (D) a ballet
-
-  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
-  Berger describes social reality using the metaphor of a puppet theatre. The answer
-  is (C).
-
-
-  Q: The term ''hegemony'' refers to:
-
-  (A) the tendency for the working class not to realize their own interests (B) a
-  dominant ideology that legitimates economic, political and cultural power (C) a
-  form of dual consciousness based on ideology and everyday experiences (D) a mode
-  of payment given for outstanding topiary
-
-  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
-  Hegemony refers to a dominant ideology that legitimates economic, policital, and
-  cultural power. The answer is (B).
-
-
-  Q: The shift from ''civil religion'' to ''common religion'' means that:
-
-  (A) the increasing bureaucracy of the state has made religion only a marginal part
-  of our lives (B) despite the weakening of traditional authority, our everyday lives
-  and ''common sense'' remain shaped by religious beliefs and values (C) religious
-  participation in collective worship may have declined, but people still practise
-  their faiths in private (D) people are much more likely to discuss their religious
-  beliefs in public, informal settings
-
-  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
-  The shift from civil religion to common religion means that despite the weakening
-  of traditional authority, our everyday lives and common sense remain shaped by religious
-  beliefs and values. The answer is (B).
-
-
-  Q: Which of the following did the post-war welfare state of 1948 not aim to provide:
-
-  (A) free health care and education for all (B) a minimum wage (C) full employment
-  (D) universal welfare
-
-  A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
-  The post-war welfare state of 1948 aimed to provide free healthcare and education,
-  full employment, and universal welfare. But it did not aim to provide a minimum
-  wage. The answer is (B).'
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_sociology
+"dataset_name": "sociology"
+"description": "The following are multiple choice questions (with answers) about sociology.\n\
+  \nQ: Which of the following is not a problem associated with official statistics\
+  \ on strike action?\n(A) most strikes go unnoticed by employers and the mass media\
+  \ (B) not all industrial disputes will be reported by the employer (C) the definition\
+  \ of strikes excludes those that involve fewer than ten workers or last less than\
+  \ one day (D) it is hard to compare strikes that were measured in different ways\n\
+  A: Let's think step by step. We refer to Wikipedia articles on sociology for help.\
+  \ Official statistics on strike action can be problematic because not all industrial\
+  \ disputes will be reported by employers, the definition of strikes excludes those\
+  \ that involves fewer than ten workers or last less than one day, and it is hard\
+  \ to compare strikes that were measured in different ways. Thus, (A) is not a problem\
+  \ associated with official statistics on strike action. The answer is (A).\n\nQ:\
+  \ What does Berger (1963) describe as a metaphor for social reality?\n(A) a fairground\
+  \ ride (B) a circus (C) a puppet theatre (D) a ballet\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on sociology for help. Berger describes social\
+  \ reality using the metaphor of a puppet theatre. The answer is (C).\n\nQ: The term\
+  \ 'hegemony' refers to:\n(A) the tendency for the working class not to realize their\
+  \ own interests (B) a dominant ideology that legitimates economic, political and\
+  \ cultural power (C) a form of dual consciousness based on ideology and everyday\
+  \ experiences (D) a mode of payment given for outstanding topiary\nA: Let's think\
+  \ step by step. We refer to Wikipedia articles on sociology for help. Hegemony refers\
+  \ to a dominant ideology that legitimates economic, policital, and cultural power.\
+  \ The answer is (B).\n\nQ: The shift from 'civil religion' to 'common religion'\
+  \ means that:\n(A) the increasing bureaucracy of the state has made religion only\
+  \ a marginal part of our lives (B) despite the weakening of traditional authority,\
+  \ our everyday lives and 'common sense' remain shaped by religious beliefs and values\
+  \ (C) religious participation in collective worship may have declined, but people\
+  \ still practise their faiths in private (D) people are much more likely to discuss\
+  \ their religious beliefs in public, informal settings\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on sociology for help. The shift from civil religion\
+  \ to common religion means that despite the weakening of traditional authority,\
+  \ our everyday lives and common sense remain shaped by religious beliefs and values.\
+  \ The answer is (B).\n\nQ: Which of the following did the post-war welfare state\
+  \ of 1948 not aim to provide:\n(A) free health care and education for all (B) a\
+  \ minimum wage (C) full employment (D) universal welfare\nA: Let's think step by\
+  \ step. We refer to Wikipedia articles on sociology for help. The post-war welfare\
+  \ state of 1948 aimed to provide free healthcare and education, full employment,\
+  \ and universal welfare. But it did not aim to provide a minimum wage. The answer\
+  \ is (B)."
+"group": "mmlu_flan_cot_fewshot_social_sciences"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_sociology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml

-dataset_name: us_foreign_policy
-description: 'The following are multiple choice questions (with answers) about us
-  foreign policy.
-
-
-  Q: How did Donald Trump attack globalization in the 2016 campaign?
-
-  (A) Globalization had made men like him too rich (B) Globalization only benefited
-  certain American states, such as New York (C) Liberal elites had encouraged globalization,
-  while ''ordinary Americans'' lost jobs because of it (D) Globalization encouraged
-  damaging trade wars
-
-  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
-  for help. Trump attacked globalization because he believed ordinary Americans lost
-  jobs due to it, and so he wanted to blame liberals who had encouraged it. The answer
-  is (C).
-
-
-  Q: How did NSC-68 change U.S. strategy?
-
-  (A) It globalized containment. (B) It militarized containment. (C) It called for
-  the development of the hydrogen bomb. (D) All of the above
-
-  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
-  for help. NSC-68 outlined a variety of courses of action, including globalization
-  of containment, militarization of contaiment, and the development of the hydrogen
-  bomb. The answer is (D).
-
-
-  Q: How do Defensive Realism and Offensive Realism differ in their explanation of
-  state behaviour?
-
-  (A) Defensive realists place greater emphasis on the role of international institutions
-  (B) Defensive realists place less emphasis on geographical factors (C) Offensive
-  realists give more priority to the national interest than Defensive realists. (D)
-  Defensive realists believe states are security maximizers, while Offensive realists
-  believe states to be power maximizers
-
-  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
-  for help. While defensive realism advocates that states are security maximizers,
-  offensive realists think of states as power maximizers. The answer is (D).
-
-
-  Q: The realm of policy decisions concerned primarily with relations between the
-  United States and the rest of the world is known as
-
-  (A) terrorism policy. (B) economic policy. (C) foreign policy. (D) international
-  policy.
-
-  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
-  for help. The topic of policy decisions concerns with relations between the US and
-  the rest of the world is known as foreign policy. The answer is (C).
-
-
-  Q: How did the 2008 financial crisis affect America''s international reputation?
-
-  (A) It damaged support for the US model of political economy and capitalism (B)
-  It created anger at the United States for exaggerating the crisis (C) It increased
-  support for American global leadership under President Obama (D) It reduced global
-  use of the US dollar
-
-  A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
-  for help. The 2008 financial crisis damanged the international reputation of the
-  American model of political economy and capitalism. The answer is (A).'
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_us_foreign_policy
+"dataset_name": "us_foreign_policy"
+"description": "The following are multiple choice questions (with answers) about us\
+  \ foreign policy.\n\nQ: How did Donald Trump attack globalization in the 2016 campaign?\n\
+  (A) Globalization had made men like him too rich (B) Globalization only benefited\
+  \ certain American states, such as New York (C) Liberal elites had encouraged globalization,\
+  \ while 'ordinary Americans' lost jobs because of it (D) Globalization encouraged\
+  \ damaging trade wars\nA: Let's think step by step. We refer to Wikipedia articles\
+  \ on us foreign policy for help. Trump attacked globalization because he believed\
+  \ ordinary Americans lost jobs due to it, and so he wanted to blame liberals who\
+  \ had encouraged it. The answer is (C).\n\nQ: How did NSC-68 change U.S. strategy?\n\
+  (A) It globalized containment. (B) It militarized containment. (C) It called for\
+  \ the development of the hydrogen bomb. (D) All of the above\nA: Let's think step\
+  \ by step. We refer to Wikipedia articles on us foreign policy for help. NSC-68\
+  \ outlined a variety of courses of action, including globalization of containment,\
+  \ militarization of contaiment, and the development of the hydrogen bomb. The answer\
+  \ is (D).\n\nQ: How do Defensive Realism and Offensive Realism differ in their explanation\
+  \ of state behaviour?\n(A) Defensive realists place greater emphasis on the role\
+  \ of international institutions (B) Defensive realists place less emphasis on geographical\
+  \ factors (C) Offensive realists give more priority to the national interest than\
+  \ Defensive realists. (D) Defensive realists believe states are security maximizers,\
+  \ while Offensive realists believe states to be power maximizers\nA: Let's think\
+  \ step by step. We refer to Wikipedia articles on us foreign policy for help. While\
+  \ defensive realism advocates that states are security maximizers, offensive realists\
+  \ think of states as power maximizers. The answer is (D).\n\nQ: The realm of policy\
+  \ decisions concerned primarily with relations between the United States and the\
+  \ rest of the world is known as\n(A) terrorism policy. (B) economic policy. (C)\
+  \ foreign policy. (D) international policy.\nA: Let's think step by step. We refer\
+  \ to Wikipedia articles on us foreign policy for help. The topic of policy decisions\
+  \ concerns with relations between the US and the rest of the world is known as foreign\
+  \ policy. The answer is (C).\n\nQ: How did the 2008 financial crisis affect America's\
+  \ international reputation?\n(A) It damaged support for the US model of political\
+  \ economy and capitalism (B) It created anger at the United States for exaggerating\
+  \ the crisis (C) It increased support for American global leadership under President\
+  \ Obama (D) It reduced global use of the US dollar\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on us foreign policy for help. The 2008 financial\
+  \ crisis damanged the international reputation of the American model of political\
+  \ economy and capitalism. The answer is (A)."
+"group": "mmlu_flan_cot_fewshot_social_sciences"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_us_foreign_policy"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml

-dataset_name: virology
-description: 'The following are multiple choice questions (with answers) about virology.
-
-
-  Q: The median survival time to AIDS and death was established by following:
-
-  (A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters
-  (D) High-risk seronegatives
-
-  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
-  The median survival time to AIDS and death was established as a result of the development
-  of seroconverters. The answer is (C).
-
-
-  Q: Which of the following is a morphological characteristic of the paramyxoviruses.
-
-  (A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate
-  viruses (C) Icosahedral viruses with envelope (D) Very large viruses
-
-  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
-  Paramyxoviruses are fragile viruses often visualised with RNA spewing from the inside.
-  The answer is (A).
-
-
-  Q: The most important goal of a behavioral intervention is:
-
-  (A) Change in behavior (B) Comprehensive coverage (C) Effective use of behavioral
-  theory (D) Sustained behavior change
-
-  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
-  The prim goal of a behavioral intervention is to cause sustained behavior change.
-  The answer is (D).
-
-
-  Q: A key factor facilitating the application of nested case-control studies from
-  the MACS was:
-
-  (A) Data collection (B) Establishment of a repository of biologic specimens (C)
-  Participant interest (D) Administration of the questionnaire by staff
-
-  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
-  The Multicenter AIDS Cohort Study''s use of nested case-control studies was facilitated
-  by the establishment of a repository of biologic specimens. The answer is (B).
-
-
-  Q: Why are parvoviruses a highly impactful parasite?
-
-  (A) Because they have no nucleic acid (B) They require a helper virus (C) Only replicate
-  in dividing cells (D) Can integrate into host chromosomes
-
-  A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
-  Paroviruses are highly impactful because they do not have nucleic acid. The answer
-  is (A).'
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_virology
+"dataset_name": "virology"
+"description": "The following are multiple choice questions (with answers) about virology.\n\
+  \nQ: The median survival time to AIDS and death was established by following:\n\
+  (A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters\
+  \ (D) High-risk seronegatives\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on virology for help. The median survival time to AIDS and death was\
+  \ established as a result of the development of seroconverters. The answer is (C).\n\
+  \nQ: Which of the following is a morphological characteristic of the paramyxoviruses.\n\
+  (A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate\
+  \ viruses (C) Icosahedral viruses with envelope (D) Very large viruses\nA: Let's\
+  \ think step by step. We refer to Wikipedia articles on virology for help. Paramyxoviruses\
+  \ are fragile viruses often visualised with RNA spewing from the inside. The answer\
+  \ is (A).\n\nQ: The most important goal of a behavioral intervention is:\n(A) Change\
+  \ in behavior (B) Comprehensive coverage (C) Effective use of behavioral theory\
+  \ (D) Sustained behavior change\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on virology for help. The prim goal of a behavioral intervention is to\
+  \ cause sustained behavior change. The answer is (D).\n\nQ: A key factor facilitating\
+  \ the application of nested case-control studies from the MACS was:\n(A) Data collection\
+  \ (B) Establishment of a repository of biologic specimens (C) Participant interest\
+  \ (D) Administration of the questionnaire by staff\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on virology for help. The Multicenter AIDS Cohort\
+  \ Study's use of nested case-control studies was facilitated by the establishment\
+  \ of a repository of biologic specimens. The answer is (B).\n\nQ: Why are parvoviruses\
+  \ a highly impactful parasite?\n(A) Because they have no nucleic acid (B) They require\
+  \ a helper virus (C) Only replicate in dividing cells (D) Can integrate into host\
+  \ chromosomes\nA: Let's think step by step. We refer to Wikipedia articles on virology\
+  \ for help. Paroviruses are highly impactful because they do not have nucleic acid.\
+  \ The answer is (A)."
+"group": "mmlu_flan_cot_fewshot_other"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_virology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml

-dataset_name: world_religions
-description: 'The following are multiple choice questions (with answers) about world
-  religions.
-
-
-  Q: How can the Upanishads be characterized?
-
-  (A) Ritual texts (B) Philosophical texts (C) Hymns (D) Origin stories
-
-  A: Let''s think step by step. We refer to Wikipedia articles on world religions
-  for help. The Upanishads are the most recent part of Vedas (the oldest scriptures
-  in Hinduism) and supplied the basis of later Hindu philosophy. So they are philosophical
-  texts. The answer is (B).
-
-
-  Q: What is the Second Gem in Buddhism?
-
-  (A) The Dharma (B) The Sangha (C) The Buddha (D) The Bodhisattva
-
-  A: Let''s think step by step. We refer to Wikipedia articles on world religions
-  for help. The Second Gem in Buddhism is The Dharma. The answer is (A).
-
-
-  Q: Which Japanese government promoted a kind of national cult based on the emperor
-  and his associations with kami?
-
-  (A) Honen (B) Tanaka (C) Tokugawa (D) Meiji
-
-  A: Let''s think step by step. We refer to Wikipedia articles on world religions
-  for help. The promotion of a national cult based on the emperor and his associations
-  with Kami happened during the reign of Emperor Meiji (1852-1912). The answer is
-  (D).
-
-
-  Q: In which dynasty was the "Mandate of Heaven" developed to legitimatize the new
-  rulers?
-
-  (A) Shang (B) Zhou (C) Han (D) Xia
-
-  A: Let''s think step by step. We refer to Wikipedia articles on world religions
-  for help. The "Mandate of Heaven" was developed as an ancient Chinese philosophical
-  concept during the Zhou Dynasty (1046-256 BCE). The answer is (B).
-
-
-  Q: What is the sign of the covenant for Jewish males?
-
-  (A) The rainbow (B) Circumcision (C) A son (D) Bar mitzvah
-
-  A: Let''s think step by step. We refer to Wikipedia articles on world religions
-  for help. In Judaism, the most distinctive sign of the covenant is circumcision
-  (brit milah). The answer is (B).'
-include: _mmlu_flan_cot_fewshot_template_yaml
-task: mmlu_flan_cot_fewshot_world_religions
+"dataset_name": "world_religions"
+"description": "The following are multiple choice questions (with answers) about world\
+  \ religions.\n\nQ: How can the Upanishads be characterized?\n(A) Ritual texts (B)\
+  \ Philosophical texts (C) Hymns (D) Origin stories\nA: Let's think step by step.\
+  \ We refer to Wikipedia articles on world religions for help. The Upanishads are\
+  \ the most recent part of Vedas (the oldest scriptures in Hinduism) and supplied\
+  \ the basis of later Hindu philosophy. So they are philosophical texts. The answer\
+  \ is (B).\n\nQ: What is the Second Gem in Buddhism?\n(A) The Dharma (B) The Sangha\
+  \ (C) The Buddha (D) The Bodhisattva\nA: Let's think step by step. We refer to Wikipedia\
+  \ articles on world religions for help. The Second Gem in Buddhism is The Dharma.\
+  \ The answer is (A).\n\nQ: Which Japanese government promoted a kind of national\
+  \ cult based on the emperor and his associations with kami?\n(A) Honen (B) Tanaka\
+  \ (C) Tokugawa (D) Meiji\nA: Let's think step by step. We refer to Wikipedia articles\
+  \ on world religions for help. The promotion of a national cult based on the emperor\
+  \ and his associations with Kami happened during the reign of Emperor Meiji (1852-1912).\
+  \ The answer is (D).\n\nQ: In which dynasty was the \"Mandate of Heaven\" developed\
+  \ to legitimatize the new rulers?\n(A) Shang (B) Zhou (C) Han (D) Xia\nA: Let's\
+  \ think step by step. We refer to Wikipedia articles on world religions for help.\
+  \ The \"Mandate of Heaven\" was developed as an ancient Chinese philosophical concept\
+  \ during the Zhou Dynasty (1046-256 BCE). The answer is (B).\n\nQ: What is the sign\
+  \ of the covenant for Jewish males?\n(A) The rainbow (B) Circumcision (C) A son\
+  \ (D) Bar mitzvah\nA: Let's think step by step. We refer to Wikipedia articles on\
+  \ world religions for help. In Judaism, the most distinctive sign of the covenant\
+  \ is circumcision (brit milah). The answer is (B)."
+"group": "mmlu_flan_cot_fewshot_humanities"
+"include": "_mmlu_flan_cot_fewshot_template_yaml"
+"task": "mmlu_flan_cot_fewshot_world_religions"

lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml

+group: mmlu_flan_cot_zeroshot
+task:
+  - mmlu_flan_cot_zeroshot_stem
+  - mmlu_flan_cot_zeroshot_other
+  - mmlu_flan_cot_zeroshot_social_sciences
+  - mmlu_flan_cot_zeroshot_humanities

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml

-dataset_name: abstract_algebra
-description: 'The following are multiple choice questions (with answers) about abstract
-  algebra.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_abstract_algebra
+"dataset_name": "abstract_algebra"
+"description": "The following are multiple choice questions (with answers) about abstract\
+  \ algebra.\n\n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_abstract_algebra"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml

-dataset_name: anatomy
-description: 'The following are multiple choice questions (with answers) about anatomy.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_anatomy
+"dataset_name": "anatomy"
+"description": "The following are multiple choice questions (with answers) about anatomy.\n\
+  \n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_anatomy"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml

-dataset_name: astronomy
-description: 'The following are multiple choice questions (with answers) about astronomy.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_astronomy
+"dataset_name": "astronomy"
+"description": "The following are multiple choice questions (with answers) about astronomy.\n\
+  \n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_astronomy"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml

-dataset_name: business_ethics
-description: 'The following are multiple choice questions (with answers) about business
-  ethics.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_business_ethics
+"dataset_name": "business_ethics"
+"description": "The following are multiple choice questions (with answers) about business\
+  \ ethics.\n\n"
+"group": "mmlu_flan_cot_zeroshot_other"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_business_ethics"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_clinical_knowledge.yaml

-dataset_name: clinical_knowledge
-description: 'The following are multiple choice questions (with answers) about clinical
-  knowledge.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_clinical_knowledge
+"dataset_name": "clinical_knowledge"
+"description": "The following are multiple choice questions (with answers) about clinical\
+  \ knowledge.\n\n"
+"group": "mmlu_flan_cot_zeroshot_other"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_clinical_knowledge"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_biology.yaml

-dataset_name: college_biology
-description: 'The following are multiple choice questions (with answers) about college
-  biology.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_college_biology
+"dataset_name": "college_biology"
+"description": "The following are multiple choice questions (with answers) about college\
+  \ biology.\n\n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_college_biology"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_chemistry.yaml

-dataset_name: college_chemistry
-description: 'The following are multiple choice questions (with answers) about college
-  chemistry.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_college_chemistry
+"dataset_name": "college_chemistry"
+"description": "The following are multiple choice questions (with answers) about college\
+  \ chemistry.\n\n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_college_chemistry"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_computer_science.yaml

-dataset_name: college_computer_science
-description: 'The following are multiple choice questions (with answers) about college
-  computer science.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_college_computer_science
+"dataset_name": "college_computer_science"
+"description": "The following are multiple choice questions (with answers) about college\
+  \ computer science.\n\n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_college_computer_science"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_mathematics.yaml

-dataset_name: college_mathematics
-description: 'The following are multiple choice questions (with answers) about college
-  mathematics.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_college_mathematics
+"dataset_name": "college_mathematics"
+"description": "The following are multiple choice questions (with answers) about college\
+  \ mathematics.\n\n"
+"group": "mmlu_flan_cot_zeroshot_stem"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_college_mathematics"

lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_college_medicine.yaml

-dataset_name: college_medicine
-description: 'The following are multiple choice questions (with answers) about college
-  medicine.
-
-
-  '
-include: _mmlu_flan_generative_template_yaml
-task: mmlu_flan_cot_zeroshot_college_medicine
+"dataset_name": "college_medicine"
+"description": "The following are multiple choice questions (with answers) about college\
+  \ medicine.\n\n"
+"group": "mmlu_flan_cot_zeroshot_other"
+"include": "_mmlu_flan_cot_zeroshot_template_yaml"
+"task": "mmlu_flan_cot_zeroshot_college_medicine"