Skip to content

GitLab

Commit 2184b8de authored by lintangsutawika's avatar lintangsutawika
Browse files

Merge branch 'cont-metrics' of https://github.com/EleutherAI/lm-evaluation-harness into alt_worlds

parents b1ba4e71 1522009c
Hide whitespace changes
Inline Side-by-side
Showing with 414 additions and 584 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_nutrition.yaml
View file @ 2184b8de
dataset_name: nutrition
description: 'The following are multiple choice questions (with answers) about nutrition.
Q: What is the first-line drug for patients with type 2 diabetes and obesity, as
of 2020?
(A) Acarbose (B) Metformin (C) Sulphonylureas (D) Insulin
A: Let''s think step by step. We refer to Wikipedia articles on nutrition for help.
Metformin (Fortamet, Glumetza, or others) is usually the first medication prescribed
for type 2 diabetes, as well as obesity. It works by lowering glucose production
in the liver and improving the body''s sensitivity to insulin. The answer is (B).
Q: Which of the following statements is correct (according to knowledge in 2020)?
(A) Consumers with phenylketonuria must avoid the consumption of the sweetener aspartame
(B) Consumers with phenylketonuria must avoid the consumption of the sweetener saccharin
(C) Consumers with phenylketonuria must avoid the consumption of the sweetener sucralose
(D) Consumers with phenylketonuria must avoid the consumption of the sweetener acesulfame
K
A: Let''s think step by step. We refer to Wikipedia articles on nutrition for help.
People with phenylketonuria (PKU) cannot break down the amino acid phenylalanine.
As it builds up in the blood and brain it can lead to brain damage. People with
PKU should avoid foods that are converted to phenylalanine in the body, such as
aspartame. The answer is (A).
Q: Which of the following statements about iodine is correct, as of 2020?
(A) 50% of adults consume iodine at levels below the RNI (B) Dairy products are
a poor source of iodine (C) The iodine content of organic milk is generally lower
that the level in non-organic milk (D) UK dietary reference values recommend an
increase in iodine intake in pregnancy
A: Let''s think step by step. We refer to Wikipedia articles on nutrition for help.
Organic milk usually has less iodine content than non-organic milk. The answer is
(C).
Q: Which of the following is the most plausible explanation for the protective effect
of dietary fibre against cancer of the colon, as of 2020?
(A) Propionic acid, formed during colonic fibre fermentation inhibits liver fatty
acid synthesis (B) Butyric acid, formed during colonic fibre fermentation stimulates
"silencing" of the SLC5A8 tumour suppressor gene (C) None of these options are correct
(D) Butyric acid, formed during colonic fibre fermentation stimulates anti-oxidant
defences in the colon
A: Let''s think step by step. We refer to Wikipedia articles on nutrition for help.
Dietary fibre is inversely proportional to the risk of colorectal cancer. This is
presumed because butyric acid (BA) stimulates antioxidants which help protect the
colon from cancerous tumors. The answer is (D).
Q: In a cohort study, the risk ratio of developing diabetes was 0.86 when comparing
consumers of tea (the exposed) to those who did not drink tea (the unexposed). Which
one statement is correct (according to knowledge in 2020)?
(A) The tea drinkers have lower risk of developing diabetes. (B) The tea drinkers
have higher risk of developing diabetes. (C) Based on the information given we cannot
tell if the observed difference in disease risk is the result of chance. (D) The
risk ratio is close to the value one, so there is no difference in disease risk
between the two groups.
A: Let''s think step by step. We refer to Wikipedia articles on nutrition for help.
The risk ratio is not sufficiently reduced that it could not be explained by random
chance given the studies sample size. The answer is (C).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_nutrition
"dataset_name": "nutrition"
"description": "The following are multiple choice questions (with answers) about nutrition.\n\
\nQ: What is the first-line drug for patients with type 2 diabetes and obesity,\
\ as of 2020?\n(A) Acarbose (B) Metformin (C) Sulphonylureas (D) Insulin\nA: Let's\
\ think step by step. We refer to Wikipedia articles on nutrition for help. Metformin\
\ (Fortamet, Glumetza, or others) is usually the first medication prescribed for\
\ type 2 diabetes, as well as obesity. It works by lowering glucose production in\
\ the liver and improving the body's sensitivity to insulin. The answer is (B).\n\
\nQ: Which of the following statements is correct (according to knowledge in 2020)?\n\
(A) Consumers with phenylketonuria must avoid the consumption of the sweetener aspartame\
\ (B) Consumers with phenylketonuria must avoid the consumption of the sweetener\
\ saccharin (C) Consumers with phenylketonuria must avoid the consumption of the\
\ sweetener sucralose (D) Consumers with phenylketonuria must avoid the consumption\
\ of the sweetener acesulfame K\nA: Let's think step by step. We refer to Wikipedia\
\ articles on nutrition for help. People with phenylketonuria (PKU) cannot break\
\ down the amino acid phenylalanine. As it builds up in the blood and brain it can\
\ lead to brain damage. People with PKU should avoid foods that are converted to\
\ phenylalanine in the body, such as aspartame. The answer is (A).\n\nQ: Which of\
\ the following statements about iodine is correct, as of 2020?\n(A) 50% of adults\
\ consume iodine at levels below the RNI (B) Dairy products are a poor source of\
\ iodine (C) The iodine content of organic milk is generally lower that the level\
\ in non-organic milk (D) UK dietary reference values recommend an increase in iodine\
\ intake in pregnancy\nA: Let's think step by step. We refer to Wikipedia articles\
\ on nutrition for help. Organic milk usually has less iodine content than non-organic\
\ milk. The answer is (C).\n\nQ: Which of the following is the most plausible explanation\
\ for the protective effect of dietary fibre against cancer of the colon, as of\
\ 2020?\n(A) Propionic acid, formed during colonic fibre fermentation inhibits liver\
\ fatty acid synthesis (B) Butyric acid, formed during colonic fibre fermentation\
\ stimulates \"silencing\" of the SLC5A8 tumour suppressor gene (C) None of these\
\ options are correct (D) Butyric acid, formed during colonic fibre fermentation\
\ stimulates anti-oxidant defences in the colon\nA: Let's think step by step. We\
\ refer to Wikipedia articles on nutrition for help. Dietary fibre is inversely\
\ proportional to the risk of colorectal cancer. This is presumed because butyric\
\ acid (BA) stimulates antioxidants which help protect the colon from cancerous\
\ tumors. The answer is (D).\n\nQ: In a cohort study, the risk ratio of developing\
\ diabetes was 0.86 when comparing consumers of tea (the exposed) to those who did\
\ not drink tea (the unexposed). Which one statement is correct (according to knowledge\
\ in 2020)?\n(A) The tea drinkers have lower risk of developing diabetes. (B) The\
\ tea drinkers have higher risk of developing diabetes. (C) Based on the information\
\ given we cannot tell if the observed difference in disease risk is the result\
\ of chance. (D) The risk ratio is close to the value one, so there is no difference\
\ in disease risk between the two groups.\nA: Let's think step by step. We refer\
\ to Wikipedia articles on nutrition for help. The risk ratio is not sufficiently\
\ reduced that it could not be explained by random chance given the studies sample\
\ size. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_nutrition"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_philosophy.yaml
View file @ 2184b8de
dataset_name: philosophy
description: "The following are multiple choice questions (with answers) about philosophy.\n\
"dataset_name": "philosophy"
"description": "The following are multiple choice questions (with answers) about philosophy.\n\
\nQ: The study of reality in the broadest sense, an inquiry into the elemental nature\
\ of the universe and the things in it, is known as _____.\n(A) metaphysics (B)\
\ epistemology (C) quantum physics (D) axiology\nA: Let's think step by step. We\
\ refer to Wikipedia articles on philosophy for help. Among the options, only metaphysics\
\ studies the nature of reality and existence. The answer is (A).\n\nQ: According\
\ to Moore\u2019s \u201Cideal utilitarianism,\u201D the right action is the one\
\ that brings about the greatest amount of:\n(A) pleasure. (B) happiness. (C) good.\
\ (D) virtue.\nA: Let's think step by step. We refer to Wikipedia articles on philosophy\
\ for help. Moore's \"ideal utilitarianism\" states that one's actions should maximize\
\ intrinsic goods. The answer is (C).\n\nQ: Before Tolstoy's Christian conversion,\
\ what was his perspective on the meaning of life?\n(A) optimist (B) satisfied (C)\
\ nominally religious (D) pessimist\nA: Let's think step by step. We refer to Wikipedia\
\ articles on philosophy for help. Before his conversion, Tolstoy feels that life\
\ was uncertain, which is a pessimist's point of view. The answer is (D).\n\nQ:\
\ According to d'Holbach, people always act according to _____.\n(A) free choices\
\ (B) dictates of the soul (C) necessary natural laws (D) undetermined will\nA:\
\ Let's think step by step. We refer to Wikipedia articles on philosophy for help.\
\ d'Holbach believes that people act according to necessary laws, and it proves\
\ nothing about people's free will. The answer is (C).\n\nQ: Psychological egoism\
\ is:\n(A) an ethical theory about how we ought to behave. (B) a generalization\
\ concerning the way people tend to behave. (C) a claim about human nature and the\
\ ways people are capable of behaving. (D) none of the above.\nA: Let's think step\
\ by step. We refer to Wikipedia articles on philosophy for help. Psychological\
\ egoism suggests that one behaves based on what makes one feels good, hence it\
\ is a claim about human nature and how humans are capable of behaving. The answer\
\ is (C)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_philosophy
\ to Moore’s “ideal utilitarianism,” the right action is the one that brings about\
\ the greatest amount of:\n(A) pleasure. (B) happiness. (C) good. (D) virtue.\n\
A: Let's think step by step. We refer to Wikipedia articles on philosophy for help.\
\ Moore's \"ideal utilitarianism\" states that one's actions should maximize intrinsic\
\ goods. The answer is (C).\n\nQ: Before Tolstoy's Christian conversion, what was\
\ his perspective on the meaning of life?\n(A) optimist (B) satisfied (C) nominally\
\ religious (D) pessimist\nA: Let's think step by step. We refer to Wikipedia articles\
\ on philosophy for help. Before his conversion, Tolstoy feels that life was uncertain,\
\ which is a pessimist's point of view. The answer is (D).\n\nQ: According to d'Holbach,\
\ people always act according to _____.\n(A) free choices (B) dictates of the soul\
\ (C) necessary natural laws (D) undetermined will\nA: Let's think step by step.\
\ We refer to Wikipedia articles on philosophy for help. d'Holbach believes that\
\ people act according to necessary laws, and it proves nothing about people's free\
\ will. The answer is (C).\n\nQ: Psychological egoism is:\n(A) an ethical theory\
\ about how we ought to behave. (B) a generalization concerning the way people tend\
\ to behave. (C) a claim about human nature and the ways people are capable of behaving.\
\ (D) none of the above.\nA: Let's think step by step. We refer to Wikipedia articles\
\ on philosophy for help. Psychological egoism suggests that one behaves based on\
\ what makes one feels good, hence it is a claim about human nature and how humans\
\ are capable of behaving. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_philosophy"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_prehistory.yaml
View file @ 2184b8de
dataset_name: prehistory
description: 'The following are multiple choice questions (with answers) about prehistory.
Q: What is the approximate mean cranial capacity of Homo erectus?
(A) under 650 cc (B) about 800 cc (C) just under 1000 cc (D) 1200 cc
A: Let''s think step by step. We refer to Wikipedia articles on prehistory for help.
The average cranium capacity of Homo erectus is less than 1000 cubic cm. The answer
is (C).
Q: According to Timothy Pauketat, the evidence for social stratification and political
power at Cahokia suggests:
(A) a center of Mississippian civilization with conditions similar to the rise of
early states. (B) the limitations of authority in a Native American society of egalitarian
foragers. (C) a simple chiefdom or perhaps a complex chiefdom had evolved by A.D.
1500. (D) a center of Mississippian civilization with conditions similar to societies
on the Northwest Coast of North America.
A: Let''s think step by step. We refer to Wikipedia articles on prehistory for help.
Timothy Pauketat is known for his research on Cahokia, the center of the Mississippian
culture, where he found similar conditions to the rise of early states. The answer
is (A).
Q: Recent research on hominid species dating from the Middle Pliocene indicates
there was (as of 2020):
(A) a great amount of species diversity, or a single species that exhibited a lot
of diversity. (B) very little species diversity during this period and very few
hominids. (C) decreased species diversity due to a prolonged ice age followed by
a severe drought. (D) decreased species diversity but increased numbers of hammerstones
and flakes, indicating stone tool manufacture.
A: Let''s think step by step. We refer to Wikipedia articles on prehistory for help.
Recent research has recognized multiple hominid species from the Middle Pliocene,
meaning that there is a great amount of species diversity or diversity in a single
species. The answer is (A).
Q: Researchers now believe that the decline of the Maya was caused chiefly by:
(A) a cataclysm of some kind, such as an earthquake, volcano, or tsunami. (B) ecological
degradation resulting from slash-and-burn farming techniques. (C) endless wars between
neighboring Mayan city-states. (D) practices of interbreeding that led to a steep
rise in congenital disorders.
A: Let''s think step by step. We refer to Wikipedia articles on prehistory for help.
Researchers believe that the Maya collapse was mainly caused by over-exploitation
of natural resources like the slash-and-burn farming techniques. The answer is (B).
Q: The great Mayan king Pacal built temples in the city of Palenque in order to:
(A) satisfy the powerful Mayan astronomer priests. (B) display his generosity to
the common people, since they were allowed to live in the temples. (C) frighten
away enemies, in particular the Spaniards. (D) legitimize his kingship, since his
father was not royal.
A: Let''s think step by step. We refer to Wikipedia articles on prehistory for help.
Pacal built the temples as the funerary monument to legitimize his kingship. The
answer is (D).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_prehistory
"dataset_name": "prehistory"
"description": "The following are multiple choice questions (with answers) about prehistory.\n\
\nQ: What is the approximate mean cranial capacity of Homo erectus?\n(A) under 650\
\ cc (B) about 800 cc (C) just under 1000 cc (D) 1200 cc\nA: Let's think step by\
\ step. We refer to Wikipedia articles on prehistory for help. The average cranium\
\ capacity of Homo erectus is less than 1000 cubic cm. The answer is (C).\n\nQ:\
\ According to Timothy Pauketat, the evidence for social stratification and political\
\ power at Cahokia suggests:\n(A) a center of Mississippian civilization with conditions\
\ similar to the rise of early states. (B) the limitations of authority in a Native\
\ American society of egalitarian foragers. (C) a simple chiefdom or perhaps a complex\
\ chiefdom had evolved by A.D. 1500. (D) a center of Mississippian civilization\
\ with conditions similar to societies on the Northwest Coast of North America.\n\
A: Let's think step by step. We refer to Wikipedia articles on prehistory for help.\
\ Timothy Pauketat is known for his research on Cahokia, the center of the Mississippian\
\ culture, where he found similar conditions to the rise of early states. The answer\
\ is (A).\n\nQ: Recent research on hominid species dating from the Middle Pliocene\
\ indicates there was (as of 2020):\n(A) a great amount of species diversity, or\
\ a single species that exhibited a lot of diversity. (B) very little species diversity\
\ during this period and very few hominids. (C) decreased species diversity due\
\ to a prolonged ice age followed by a severe drought. (D) decreased species diversity\
\ but increased numbers of hammerstones and flakes, indicating stone tool manufacture.\n\
A: Let's think step by step. We refer to Wikipedia articles on prehistory for help.\
\ Recent research has recognized multiple hominid species from the Middle Pliocene,\
\ meaning that there is a great amount of species diversity or diversity in a single\
\ species. The answer is (A).\n\nQ: Researchers now believe that the decline of\
\ the Maya was caused chiefly by:\n(A) a cataclysm of some kind, such as an earthquake,\
\ volcano, or tsunami. (B) ecological degradation resulting from slash-and-burn\
\ farming techniques. (C) endless wars between neighboring Mayan city-states. (D)\
\ practices of interbreeding that led to a steep rise in congenital disorders.\n\
A: Let's think step by step. We refer to Wikipedia articles on prehistory for help.\
\ Researchers believe that the Maya collapse was mainly caused by over-exploitation\
\ of natural resources like the slash-and-burn farming techniques. The answer is\
\ (B).\n\nQ: The great Mayan king Pacal built temples in the city of Palenque in\
\ order to:\n(A) satisfy the powerful Mayan astronomer priests. (B) display his\
\ generosity to the common people, since they were allowed to live in the temples.\
\ (C) frighten away enemies, in particular the Spaniards. (D) legitimize his kingship,\
\ since his father was not royal.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on prehistory for help. Pacal built the temples as the funerary monument\
\ to legitimize his kingship. The answer is (D)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_prehistory"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_accounting.yaml
View file @ 2184b8de
dataset_name: professional_accounting
description: "The following are multiple choice questions (with answers) about professional\
\ accounting.\n\nQ: An auditor traces the serial numbers on equipment to a nonissuer\u2019\
s subledger. Which of the following management assertions is supported by this test?\n\
"dataset_name": "professional_accounting"
"description": "The following are multiple choice questions (with answers) about professional\
\ accounting.\n\nQ: An auditor traces the serial numbers on equipment to a nonissuer’s\
\ subledger. Which of the following management assertions is supported by this test?\n\
(A) Valuation and allocation (B) Completeness (C) Rights and obligations (D) Presentation\
\ and disclosure\nA: Let's think step by step. We refer to Wikipedia articles on\
\ accounting for help. The completeness assertion is tested by tracing supporting\
......@@ -43,5 +43,6 @@ description: "The following are multiple choice questions (with answers) about p
\ transactions, only Proceeds from long-term debt belongs to the financing activities\
\ section of cashflow, hence the amount reported should be $100000. The answer is\
\ (D)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_professional_accounting
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_professional_accounting"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_law.yaml
View file @ 2184b8de
dataset_name: professional_law
description: "The following are multiple choice questions (with answers) about professional\
"dataset_name": "professional_law"
"description": "The following are multiple choice questions (with answers) about professional\
\ law.\n\nQ: A son owed a creditor $5,000. The son's father contacted the creditor\
\ and told him that he wanted to pay the son's debt. The father signed a document\
\ that stated the father would pay the son's debt at a rate of $500 a month for\
......@@ -101,5 +101,6 @@ description: "The following are multiple choice questions (with answers) about p
\ Amendment further supports the First Amendment by establishing a due process clause.\
\ Hence the strongest argument should be the statute is overbroad and consequently\
\ invalid under the First and Fourteenth Amendments. The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_professional_law
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_professional_law"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_medicine.yaml
View file @ 2184b8de
dataset_name: professional_medicine
description: "The following are multiple choice questions (with answers) about professional\
"dataset_name": "professional_medicine"
"description": "The following are multiple choice questions (with answers) about professional\
\ medicine.\n\nQ: A 22-year-old male marathon runner presents to the office with\
\ the complaint of right-sided rib pain when he runs long distances. Physical examination\
\ reveals normal heart and lung findings and an exhalation dysfunction at ribs\_\
4-5 on the right. Which of the following muscles or muscle groups will be most useful\
\ reveals normal heart and lung findings and an exhalation dysfunction at ribs 4-5\
\ on the right. Which of the following muscles or muscle groups will be most useful\
\ in correcting this dysfunction utilizing a direct method?\n(A) anterior scalene\
\ (B) latissimus dorsi (C) pectoralis minor (D) quadratus lumborum\nA: Let's think\
\ step by step. We refer to Wikipedia articles on medicine for help. Among the options,\
\ only pectoralis minor muscle origins from the outer surfaces of the 3rd to 5th\
\ ribs. The answer is (C).\n\nQ: A 36-year-old male presents to the office with\
\ a\_3-week\_history of low back pain. He denies any recent trauma but says that\
\ he climbs in and out of his truck numerous times a day for his job. Examination\
\ of the patient in the prone position reveals a deep sacral sulcus on the left,\
\ a posterior inferior lateral angle on the right, and a lumbosacral junction that\
\ a 3-week history of low back pain. He denies any recent trauma but says that he\
\ climbs in and out of his truck numerous times a day for his job. Examination of\
\ the patient in the prone position reveals a deep sacral sulcus on the left, a\
\ posterior inferior lateral angle on the right, and a lumbosacral junction that\
\ springs freely on compression. The most likely diagnosis is\n(A) left-on-left\
\ sacral torsion (B) left-on-right sacral torsion (C) right unilateral sacral flexion\
\ (D) right-on-right sacral torsion\nA: Let's think step by step. We refer to Wikipedia\
......@@ -23,9 +23,9 @@ description: "The following are multiple choice questions (with answers) about p
\ nonproductive cough, runny nose, and frontal headache. He says the headache is\
\ worse in the morning and ibuprofen does provide some relief. He has not had shortness\
\ of breath. Medical history is unremarkable. He takes no medications other than\
\ the ibuprofen for pain. Vital signs are temperature 37.4\xB0C (99.4\xB0F), pulse\
\ 88/min, respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the\
\ nares shows erythematous mucous membranes. Examination of the throat shows erythema\
\ the ibuprofen for pain. Vital signs are temperature 37.4°C (99.4°F), pulse 88/min,\
\ respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the nares\
\ shows erythematous mucous membranes. Examination of the throat shows erythema\
\ and follicular lymphoid hyperplasia on the posterior oropharynx. There is no palpable\
\ cervical adenopathy. Lungs are clear to auscultation. Which of the following is\
\ the most likely cause of this patient's symptoms?\n(A) Allergic rhinitis (B) Epstein-Barr\
......@@ -57,13 +57,14 @@ description: "The following are multiple choice questions (with answers) about p
\ A follow-up visit in the office 2 weeks ago disclosed elevated urinary normetanephrine\
\ and metanephrine and plasma aldosterone concentrations. The patient was referred\
\ to a surgeon, who recommended the adrenalectomy. Today, vital signs are temperature\
\ 36.6\xB0C (97.9\xB0F), pulse 100/min, respirations 14/min, and blood pressure\
\ 170/95 mm Hg. Physical examination discloses no significant findings. Initial\
\ preoperative preparation should include treatment with which of the following?\n\
(A) Labetalol (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxybenzamine\n\
\ 36.6°C (97.9°F), pulse 100/min, respirations 14/min, and blood pressure 170/95\
\ mm Hg. Physical examination discloses no significant findings. Initial preoperative\
\ preparation should include treatment with which of the following?\n(A) Labetalol\
\ (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxybenzamine\n\
A: Let's think step by step. We refer to Wikipedia articles on medicine for help.\
\ The symptoms and the adrenal mass suggested pheochromocytoma, and the blood pressure\
\ indicates hypertension. Phenoxybenzamine is used to treat hypertension caused\
\ by pheochromocytoma. The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_professional_medicine
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_professional_medicine"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_professional_psychology.yaml
View file @ 2184b8de
dataset_name: professional_psychology
description: "The following are multiple choice questions (with answers) about professional\
"dataset_name": "professional_psychology"
"description": "The following are multiple choice questions (with answers) about professional\
\ psychology.\n\nQ: In the construction of a multiple regression equation for purposes\
\ of prediction, the optimal combination of measures is one in which the predictors\n\
(A) are uncorrelated with each other but are moderately correlated with the criterion\
......@@ -18,30 +18,31 @@ description: "The following are multiple choice questions (with answers) about p
\ step by step. We refer to Wikipedia articles on psychology for help. The definition\
\ of mode is the most frequently occurring number. The answer is (D).\n\nQ: Carl\
\ Jung believed that a client's transference:\n(A) is a fantasy that distracts the\
\ client from reality. (B) represents \u201Cmixed feelings\u201D toward the therapist.\
\ (C) \"is a form of \"\"acting out.\"\"\" (D) reflects the client\u2019s personal\
\ and collective unconscious.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on psychology for help. Transference is a phenomenon that a person's\
\ feelings are unconsciously redirected, so it reflects the client's personal and\
\ collective unconscious. The answer is (D).\n\nQ: In terms of Hofstede\u2019s (1980)\
\ five cultural dimensions, the United States scores at the top of the scale on:\n\
(A) individualism. (B) individualism and power distance. (C) power distance and\
\ masculinity. (D) uncertainty avoidance.\nA: Let's think step by step. We refer\
\ to Wikipedia articles on psychology for help. US scores highest on individualism\
\ among the five cultural dimensions. The answer is (A).\n\nQ: One of your therapy\
\ clients asks your advice about a good weight- reduction program. You have investigated\
\ the programs in the community and are enrolled in the one you consider the best.\
\ This program offers a $50 bonus to its patrons for each new person they bring\
\ into the program. Under these circumstances, your most appropriate response would\
\ be to\n(A) tell your client the pros and cons of each program you know about except\
\ for the one in which you are enrolled (B) recommend to your client the program\
\ in which you are enrolled and explain the $50 bonus you will receive (C) recommend\
\ to your client the program in which you are enrolled and offer to have the $50\
\ bonus credited to your client's account in the program (D) tell your client the\
\ pros and cons of each program you know about, but do not claim the $50 bonus if\
\ your client enrolls in your program\nA: Let's think step by step. We refer to\
\ Wikipedia articles on psychology for help. Based on the circumstances, you should\
\ tell your client about the pros and cons of each program, but it would be inappropriate\
\ to receive the bonus, so you should not claim the $50 bonus. The answer is (D)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_professional_psychology
\ client from reality. (B) represents “mixed feelings” toward the therapist. (C)\
\ \"is a form of \"\"acting out.\"\"\" (D) reflects the client’s personal and collective\
\ unconscious.\nA: Let's think step by step. We refer to Wikipedia articles on psychology\
\ for help. Transference is a phenomenon that a person's feelings are unconsciously\
\ redirected, so it reflects the client's personal and collective unconscious. The\
\ answer is (D).\n\nQ: In terms of Hofstede’s (1980) five cultural dimensions, the\
\ United States scores at the top of the scale on:\n(A) individualism. (B) individualism\
\ and power distance. (C) power distance and masculinity. (D) uncertainty avoidance.\n\
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
\ US scores highest on individualism among the five cultural dimensions. The answer\
\ is (A).\n\nQ: One of your therapy clients asks your advice about a good weight-\
\ reduction program. You have investigated the programs in the community and are\
\ enrolled in the one you consider the best. This program offers a $50 bonus to\
\ its patrons for each new person they bring into the program. Under these circumstances,\
\ your most appropriate response would be to\n(A) tell your client the pros and\
\ cons of each program you know about except for the one in which you are enrolled\
\ (B) recommend to your client the program in which you are enrolled and explain\
\ the $50 bonus you will receive (C) recommend to your client the program in which\
\ you are enrolled and offer to have the $50 bonus credited to your client's account\
\ in the program (D) tell your client the pros and cons of each program you know\
\ about, but do not claim the $50 bonus if your client enrolls in your program\n\
A: Let's think step by step. We refer to Wikipedia articles on psychology for help.\
\ Based on the circumstances, you should tell your client about the pros and cons\
\ of each program, but it would be inappropriate to receive the bonus, so you should\
\ not claim the $50 bonus. The answer is (D)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_professional_psychology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml
View file @ 2184b8de
dataset_name: public_relations
description: 'The following are multiple choice questions (with answers) about public
relations.
Q: Earth Hour was a campaign launched by which organization?
(A) Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund
A: Let''s think step by step. We refer to Wikipedia articles on public relations
for help. Earth Hour is a worldwide movement oragnized launched by the World Wildlife
Fund. The answer is (D).
Q: In issues management, what is the most proactive approach to addressing negative
or misleading information posted online about your organization?
(A) Buy domain names that could be used by opposition groups. (B) Post anonymous
comments on blogs to combat this information. (C) Prepare a news release that discredits
the inaccurate information. (D) Make policy changes to address complaints highlighted
on these sites.
A: Let''s think step by step. We refer to Wikipedia articles on public relations
for help. In issues management, the most proactive approach to addressing negative
or misleading information posted online is to make policy changes to address complaints
highlighted on those sites. The answer is (D).
Q: At which stage in the planning process would a situation analysis be carried
out?
(A) Defining the program (B) Planning the program (C) Taking action and implementing
ideas (D) Evaluation of the program
A: Let''s think step by step. We refer to Wikipedia articles on public relations
for help. Situation analyses are typically carried out during the planning process
stage of defining the program. The answer is (A).
Q: Which of these statements is true of the Vatican in 2010 at the time of the accusations
of child abuse cover-ups?
(A) There was a coordinated media response. (B) Consistent messages were communicated.
(C) Criticisms were taken as attacks on the Catholic Church. (D) The credibility
of the Vatican was upheld.
A: Let''s think step by step. We refer to Wikipedia articles on public relations
for help. In 2010 when there were accusations of child abuse cover-ups, the Vatican
took those criticisms as attacks on the Catholic Church. The answer is (C).
Q: What should a public relations media practitioner do if she does not know the
answer to a reporter''s question?
(A) Give the reporter other information she is certain is correct. (B) Say that
the information is ''off the record'' and will be disseminated later. (C) Say ''I
don''t know'' and promise to provide the information later. (D) Say ''no comment,''
rather than appear uninformed.
A: Let''s think step by step. We refer to Wikipedia articles on public relations
for help. If a public relations media practitioner does not know the answer to a
reporter''s question, they should say ''I don''t know'' and offer to provide the
information later. The answer is (C).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_public_relations
"dataset_name": "public_relations"
"description": "The following are multiple choice questions (with answers) about public\
\ relations.\n\nQ: Earth Hour was a campaign launched by which organization?\n(A)\
\ Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund\nA: Let's think step by\
\ step. We refer to Wikipedia articles on public relations for help. Earth Hour\
\ is a worldwide movement oragnized launched by the World Wildlife Fund. The answer\
\ is (D).\n\nQ: In issues management, what is the most proactive approach to addressing\
\ negative or misleading information posted online about your organization?\n(A)\
\ Buy domain names that could be used by opposition groups. (B) Post anonymous comments\
\ on blogs to combat this information. (C) Prepare a news release that discredits\
\ the inaccurate information. (D) Make policy changes to address complaints highlighted\
\ on these sites.\nA: Let's think step by step. We refer to Wikipedia articles on\
\ public relations for help. In issues management, the most proactive approach to\
\ addressing negative or misleading information posted online is to make policy\
\ changes to address complaints highlighted on those sites. The answer is (D).\n\
\nQ: At which stage in the planning process would a situation analysis be carried\
\ out?\n(A) Defining the program (B) Planning the program (C) Taking action and\
\ implementing ideas (D) Evaluation of the program\nA: Let's think step by step.\
\ We refer to Wikipedia articles on public relations for help. Situation analyses\
\ are typically carried out during the planning process stage of defining the program.\
\ The answer is (A).\n\nQ: Which of these statements is true of the Vatican in 2010\
\ at the time of the accusations of child abuse cover-ups?\n(A) There was a coordinated\
\ media response. (B) Consistent messages were communicated. (C) Criticisms were\
\ taken as attacks on the Catholic Church. (D) The credibility of the Vatican was\
\ upheld.\nA: Let's think step by step. We refer to Wikipedia articles on public\
\ relations for help. In 2010 when there were accusations of child abuse cover-ups,\
\ the Vatican took those criticisms as attacks on the Catholic Church. The answer\
\ is (C).\n\nQ: What should a public relations media practitioner do if she does\
\ not know the answer to a reporter's question?\n(A) Give the reporter other information\
\ she is certain is correct. (B) Say that the information is 'off the record' and\
\ will be disseminated later. (C) Say 'I don't know' and promise to provide the\
\ information later. (D) Say 'no comment,' rather than appear uninformed.\nA: Let's\
\ think step by step. We refer to Wikipedia articles on public relations for help.\
\ If a public relations media practitioner does not know the answer to a reporter's\
\ question, they should say 'I don't know' and offer to provide the information\
\ later. The answer is (C)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_public_relations"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml
View file @ 2184b8de
dataset_name: security_studies
description: "The following are multiple choice questions (with answers) about security\
"dataset_name": "security_studies"
"description": "The following are multiple choice questions (with answers) about security\
\ studies.\n\nQ: What are the frameworks of analysis within which terrorism has\
\ been considered (as of 2020)?\n(A) Competition between larger nations has resulted\
\ in some countries actively supporting terrorist groups to undermine the strength\
......@@ -81,5 +81,6 @@ description: "The following are multiple choice questions (with answers) about s
\ for negotiation or concession.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on security studies for help. Coercive diplomacy uses the threat of force\
\ to induce the opponent to comply with demands. The answer is (B)."
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_security_studies
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_security_studies"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml
View file @ 2184b8de
dataset_name: sociology
description: 'The following are multiple choice questions (with answers) about sociology.
Q: Which of the following is not a problem associated with official statistics on
strike action?
(A) most strikes go unnoticed by employers and the mass media (B) not all industrial
disputes will be reported by the employer (C) the definition of strikes excludes
those that involve fewer than ten workers or last less than one day (D) it is hard
to compare strikes that were measured in different ways
A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
Official statistics on strike action can be problematic because not all industrial
disputes will be reported by employers, the definition of strikes excludes those
that involves fewer than ten workers or last less than one day, and it is hard to
compare strikes that were measured in different ways. Thus, (A) is not a problem
associated with official statistics on strike action. The answer is (A).
Q: What does Berger (1963) describe as a metaphor for social reality?
(A) a fairground ride (B) a circus (C) a puppet theatre (D) a ballet
A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
Berger describes social reality using the metaphor of a puppet theatre. The answer
is (C).
Q: The term ''hegemony'' refers to:
(A) the tendency for the working class not to realize their own interests (B) a
dominant ideology that legitimates economic, political and cultural power (C) a
form of dual consciousness based on ideology and everyday experiences (D) a mode
of payment given for outstanding topiary
A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
Hegemony refers to a dominant ideology that legitimates economic, policital, and
cultural power. The answer is (B).
Q: The shift from ''civil religion'' to ''common religion'' means that:
(A) the increasing bureaucracy of the state has made religion only a marginal part
of our lives (B) despite the weakening of traditional authority, our everyday lives
and ''common sense'' remain shaped by religious beliefs and values (C) religious
participation in collective worship may have declined, but people still practise
their faiths in private (D) people are much more likely to discuss their religious
beliefs in public, informal settings
A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
The shift from civil religion to common religion means that despite the weakening
of traditional authority, our everyday lives and common sense remain shaped by religious
beliefs and values. The answer is (B).
Q: Which of the following did the post-war welfare state of 1948 not aim to provide:
(A) free health care and education for all (B) a minimum wage (C) full employment
(D) universal welfare
A: Let''s think step by step. We refer to Wikipedia articles on sociology for help.
The post-war welfare state of 1948 aimed to provide free healthcare and education,
full employment, and universal welfare. But it did not aim to provide a minimum
wage. The answer is (B).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_sociology
"dataset_name": "sociology"
"description": "The following are multiple choice questions (with answers) about sociology.\n\
\nQ: Which of the following is not a problem associated with official statistics\
\ on strike action?\n(A) most strikes go unnoticed by employers and the mass media\
\ (B) not all industrial disputes will be reported by the employer (C) the definition\
\ of strikes excludes those that involve fewer than ten workers or last less than\
\ one day (D) it is hard to compare strikes that were measured in different ways\n\
A: Let's think step by step. We refer to Wikipedia articles on sociology for help.\
\ Official statistics on strike action can be problematic because not all industrial\
\ disputes will be reported by employers, the definition of strikes excludes those\
\ that involves fewer than ten workers or last less than one day, and it is hard\
\ to compare strikes that were measured in different ways. Thus, (A) is not a problem\
\ associated with official statistics on strike action. The answer is (A).\n\nQ:\
\ What does Berger (1963) describe as a metaphor for social reality?\n(A) a fairground\
\ ride (B) a circus (C) a puppet theatre (D) a ballet\nA: Let's think step by step.\
\ We refer to Wikipedia articles on sociology for help. Berger describes social\
\ reality using the metaphor of a puppet theatre. The answer is (C).\n\nQ: The term\
\ 'hegemony' refers to:\n(A) the tendency for the working class not to realize their\
\ own interests (B) a dominant ideology that legitimates economic, political and\
\ cultural power (C) a form of dual consciousness based on ideology and everyday\
\ experiences (D) a mode of payment given for outstanding topiary\nA: Let's think\
\ step by step. We refer to Wikipedia articles on sociology for help. Hegemony refers\
\ to a dominant ideology that legitimates economic, policital, and cultural power.\
\ The answer is (B).\n\nQ: The shift from 'civil religion' to 'common religion'\
\ means that:\n(A) the increasing bureaucracy of the state has made religion only\
\ a marginal part of our lives (B) despite the weakening of traditional authority,\
\ our everyday lives and 'common sense' remain shaped by religious beliefs and values\
\ (C) religious participation in collective worship may have declined, but people\
\ still practise their faiths in private (D) people are much more likely to discuss\
\ their religious beliefs in public, informal settings\nA: Let's think step by step.\
\ We refer to Wikipedia articles on sociology for help. The shift from civil religion\
\ to common religion means that despite the weakening of traditional authority,\
\ our everyday lives and common sense remain shaped by religious beliefs and values.\
\ The answer is (B).\n\nQ: Which of the following did the post-war welfare state\
\ of 1948 not aim to provide:\n(A) free health care and education for all (B) a\
\ minimum wage (C) full employment (D) universal welfare\nA: Let's think step by\
\ step. We refer to Wikipedia articles on sociology for help. The post-war welfare\
\ state of 1948 aimed to provide free healthcare and education, full employment,\
\ and universal welfare. But it did not aim to provide a minimum wage. The answer\
\ is (B)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_sociology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml
View file @ 2184b8de
dataset_name: us_foreign_policy
description: 'The following are multiple choice questions (with answers) about us
foreign policy.
Q: How did Donald Trump attack globalization in the 2016 campaign?
(A) Globalization had made men like him too rich (B) Globalization only benefited
certain American states, such as New York (C) Liberal elites had encouraged globalization,
while ''ordinary Americans'' lost jobs because of it (D) Globalization encouraged
damaging trade wars
A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
for help. Trump attacked globalization because he believed ordinary Americans lost
jobs due to it, and so he wanted to blame liberals who had encouraged it. The answer
is (C).
Q: How did NSC-68 change U.S. strategy?
(A) It globalized containment. (B) It militarized containment. (C) It called for
the development of the hydrogen bomb. (D) All of the above
A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
for help. NSC-68 outlined a variety of courses of action, including globalization
of containment, militarization of contaiment, and the development of the hydrogen
bomb. The answer is (D).
Q: How do Defensive Realism and Offensive Realism differ in their explanation of
state behaviour?
(A) Defensive realists place greater emphasis on the role of international institutions
(B) Defensive realists place less emphasis on geographical factors (C) Offensive
realists give more priority to the national interest than Defensive realists. (D)
Defensive realists believe states are security maximizers, while Offensive realists
believe states to be power maximizers
A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
for help. While defensive realism advocates that states are security maximizers,
offensive realists think of states as power maximizers. The answer is (D).
Q: The realm of policy decisions concerned primarily with relations between the
United States and the rest of the world is known as
(A) terrorism policy. (B) economic policy. (C) foreign policy. (D) international
policy.
A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
for help. The topic of policy decisions concerns with relations between the US and
the rest of the world is known as foreign policy. The answer is (C).
Q: How did the 2008 financial crisis affect America''s international reputation?
(A) It damaged support for the US model of political economy and capitalism (B)
It created anger at the United States for exaggerating the crisis (C) It increased
support for American global leadership under President Obama (D) It reduced global
use of the US dollar
A: Let''s think step by step. We refer to Wikipedia articles on us foreign policy
for help. The 2008 financial crisis damanged the international reputation of the
American model of political economy and capitalism. The answer is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_us_foreign_policy
"dataset_name": "us_foreign_policy"
"description": "The following are multiple choice questions (with answers) about us\
\ foreign policy.\n\nQ: How did Donald Trump attack globalization in the 2016 campaign?\n\
(A) Globalization had made men like him too rich (B) Globalization only benefited\
\ certain American states, such as New York (C) Liberal elites had encouraged globalization,\
\ while 'ordinary Americans' lost jobs because of it (D) Globalization encouraged\
\ damaging trade wars\nA: Let's think step by step. We refer to Wikipedia articles\
\ on us foreign policy for help. Trump attacked globalization because he believed\
\ ordinary Americans lost jobs due to it, and so he wanted to blame liberals who\
\ had encouraged it. The answer is (C).\n\nQ: How did NSC-68 change U.S. strategy?\n\
(A) It globalized containment. (B) It militarized containment. (C) It called for\
\ the development of the hydrogen bomb. (D) All of the above\nA: Let's think step\
\ by step. We refer to Wikipedia articles on us foreign policy for help. NSC-68\
\ outlined a variety of courses of action, including globalization of containment,\
\ militarization of contaiment, and the development of the hydrogen bomb. The answer\
\ is (D).\n\nQ: How do Defensive Realism and Offensive Realism differ in their explanation\
\ of state behaviour?\n(A) Defensive realists place greater emphasis on the role\
\ of international institutions (B) Defensive realists place less emphasis on geographical\
\ factors (C) Offensive realists give more priority to the national interest than\
\ Defensive realists. (D) Defensive realists believe states are security maximizers,\
\ while Offensive realists believe states to be power maximizers\nA: Let's think\
\ step by step. We refer to Wikipedia articles on us foreign policy for help. While\
\ defensive realism advocates that states are security maximizers, offensive realists\
\ think of states as power maximizers. The answer is (D).\n\nQ: The realm of policy\
\ decisions concerned primarily with relations between the United States and the\
\ rest of the world is known as\n(A) terrorism policy. (B) economic policy. (C)\
\ foreign policy. (D) international policy.\nA: Let's think step by step. We refer\
\ to Wikipedia articles on us foreign policy for help. The topic of policy decisions\
\ concerns with relations between the US and the rest of the world is known as foreign\
\ policy. The answer is (C).\n\nQ: How did the 2008 financial crisis affect America's\
\ international reputation?\n(A) It damaged support for the US model of political\
\ economy and capitalism (B) It created anger at the United States for exaggerating\
\ the crisis (C) It increased support for American global leadership under President\
\ Obama (D) It reduced global use of the US dollar\nA: Let's think step by step.\
\ We refer to Wikipedia articles on us foreign policy for help. The 2008 financial\
\ crisis damanged the international reputation of the American model of political\
\ economy and capitalism. The answer is (A)."
"group": "mmlu_flan_cot_fewshot_social_sciences"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_us_foreign_policy"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml
View file @ 2184b8de
dataset_name: virology
description: 'The following are multiple choice questions (with answers) about virology.
Q: The median survival time to AIDS and death was established by following:
(A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters
(D) High-risk seronegatives
A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
The median survival time to AIDS and death was established as a result of the development
of seroconverters. The answer is (C).
Q: Which of the following is a morphological characteristic of the paramyxoviruses.
(A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate
viruses (C) Icosahedral viruses with envelope (D) Very large viruses
A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
Paramyxoviruses are fragile viruses often visualised with RNA spewing from the inside.
The answer is (A).
Q: The most important goal of a behavioral intervention is:
(A) Change in behavior (B) Comprehensive coverage (C) Effective use of behavioral
theory (D) Sustained behavior change
A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
The prim goal of a behavioral intervention is to cause sustained behavior change.
The answer is (D).
Q: A key factor facilitating the application of nested case-control studies from
the MACS was:
(A) Data collection (B) Establishment of a repository of biologic specimens (C)
Participant interest (D) Administration of the questionnaire by staff
A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
The Multicenter AIDS Cohort Study''s use of nested case-control studies was facilitated
by the establishment of a repository of biologic specimens. The answer is (B).
Q: Why are parvoviruses a highly impactful parasite?
(A) Because they have no nucleic acid (B) They require a helper virus (C) Only replicate
in dividing cells (D) Can integrate into host chromosomes
A: Let''s think step by step. We refer to Wikipedia articles on virology for help.
Paroviruses are highly impactful because they do not have nucleic acid. The answer
is (A).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_virology
"dataset_name": "virology"
"description": "The following are multiple choice questions (with answers) about virology.\n\
\nQ: The median survival time to AIDS and death was established by following:\n\
(A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters\
\ (D) High-risk seronegatives\nA: Let's think step by step. We refer to Wikipedia\
\ articles on virology for help. The median survival time to AIDS and death was\
\ established as a result of the development of seroconverters. The answer is (C).\n\
\nQ: Which of the following is a morphological characteristic of the paramyxoviruses.\n\
(A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate\
\ viruses (C) Icosahedral viruses with envelope (D) Very large viruses\nA: Let's\
\ think step by step. We refer to Wikipedia articles on virology for help. Paramyxoviruses\
\ are fragile viruses often visualised with RNA spewing from the inside. The answer\
\ is (A).\n\nQ: The most important goal of a behavioral intervention is:\n(A) Change\
\ in behavior (B) Comprehensive coverage (C) Effective use of behavioral theory\
\ (D) Sustained behavior change\nA: Let's think step by step. We refer to Wikipedia\
\ articles on virology for help. The prim goal of a behavioral intervention is to\
\ cause sustained behavior change. The answer is (D).\n\nQ: A key factor facilitating\
\ the application of nested case-control studies from the MACS was:\n(A) Data collection\
\ (B) Establishment of a repository of biologic specimens (C) Participant interest\
\ (D) Administration of the questionnaire by staff\nA: Let's think step by step.\
\ We refer to Wikipedia articles on virology for help. The Multicenter AIDS Cohort\
\ Study's use of nested case-control studies was facilitated by the establishment\
\ of a repository of biologic specimens. The answer is (B).\n\nQ: Why are parvoviruses\
\ a highly impactful parasite?\n(A) Because they have no nucleic acid (B) They require\
\ a helper virus (C) Only replicate in dividing cells (D) Can integrate into host\
\ chromosomes\nA: Let's think step by step. We refer to Wikipedia articles on virology\
\ for help. Paroviruses are highly impactful because they do not have nucleic acid.\
\ The answer is (A)."
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_virology"
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml
View file @ 2184b8de
dataset_name: world_religions
description: 'The following are multiple choice questions (with answers) about world
religions.
Q: How can the Upanishads be characterized?
(A) Ritual texts (B) Philosophical texts (C) Hymns (D) Origin stories
A: Let''s think step by step. We refer to Wikipedia articles on world religions
for help. The Upanishads are the most recent part of Vedas (the oldest scriptures
in Hinduism) and supplied the basis of later Hindu philosophy. So they are philosophical
texts. The answer is (B).
Q: What is the Second Gem in Buddhism?
(A) The Dharma (B) The Sangha (C) The Buddha (D) The Bodhisattva
A: Let''s think step by step. We refer to Wikipedia articles on world religions
for help. The Second Gem in Buddhism is The Dharma. The answer is (A).
Q: Which Japanese government promoted a kind of national cult based on the emperor
and his associations with kami?
(A) Honen (B) Tanaka (C) Tokugawa (D) Meiji
A: Let''s think step by step. We refer to Wikipedia articles on world religions
for help. The promotion of a national cult based on the emperor and his associations
with Kami happened during the reign of Emperor Meiji (1852-1912). The answer is
(D).
Q: In which dynasty was the "Mandate of Heaven" developed to legitimatize the new
rulers?
(A) Shang (B) Zhou (C) Han (D) Xia
A: Let''s think step by step. We refer to Wikipedia articles on world religions
for help. The "Mandate of Heaven" was developed as an ancient Chinese philosophical
concept during the Zhou Dynasty (1046-256 BCE). The answer is (B).
Q: What is the sign of the covenant for Jewish males?
(A) The rainbow (B) Circumcision (C) A son (D) Bar mitzvah
A: Let''s think step by step. We refer to Wikipedia articles on world religions
for help. In Judaism, the most distinctive sign of the covenant is circumcision
(brit milah). The answer is (B).'
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_world_religions
"dataset_name": "world_religions"
"description": "The following are multiple choice questions (with answers) about world\
\ religions.\n\nQ: How can the Upanishads be characterized?\n(A) Ritual texts (B)\
\ Philosophical texts (C) Hymns (D) Origin stories\nA: Let's think step by step.\
\ We refer to Wikipedia articles on world religions for help. The Upanishads are\
\ the most recent part of Vedas (the oldest scriptures in Hinduism) and supplied\
\ the basis of later Hindu philosophy. So they are philosophical texts. The answer\
\ is (B).\n\nQ: What is the Second Gem in Buddhism?\n(A) The Dharma (B) The Sangha\
\ (C) The Buddha (D) The Bodhisattva\nA: Let's think step by step. We refer to Wikipedia\
\ articles on world religions for help. The Second Gem in Buddhism is The Dharma.\
\ The answer is (A).\n\nQ: Which Japanese government promoted a kind of national\
\ cult based on the emperor and his associations with kami?\n(A) Honen (B) Tanaka\
\ (C) Tokugawa (D) Meiji\nA: Let's think step by step. We refer to Wikipedia articles\
\ on world religions for help. The promotion of a national cult based on the emperor\
\ and his associations with Kami happened during the reign of Emperor Meiji (1852-1912).\
\ The answer is (D).\n\nQ: In which dynasty was the \"Mandate of Heaven\" developed\
\ to legitimatize the new rulers?\n(A) Shang (B) Zhou (C) Han (D) Xia\nA: Let's\
\ think step by step. We refer to Wikipedia articles on world religions for help.\
\ The \"Mandate of Heaven\" was developed as an ancient Chinese philosophical concept\
\ during the Zhou Dynasty (1046-256 BCE). The answer is (B).\n\nQ: What is the sign\
\ of the covenant for Jewish males?\n(A) The rainbow (B) Circumcision (C) A son\
\ (D) Bar mitzvah\nA: Let's think step by step. We refer to Wikipedia articles on\
\ world religions for help. In Judaism, the most distinctive sign of the covenant\
\ is circumcision (brit milah). The answer is (B)."
"group": "mmlu_flan_cot_fewshot_humanities"
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_world_religions"
lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml 0 → 100644
View file @ 2184b8de
group: mmlu_flan_cot_zeroshot
task:
- mmlu_flan_cot_zeroshot_stem
- mmlu_flan_cot_zeroshot_other
- mmlu_flan_cot_zeroshot_social_sciences
- mmlu_flan_cot_zeroshot_humanities
lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_cot_zeroshot_template_yaml 0 → 100644
View file @ 2184b8de
group: mmlu_flan_cot_zeroshot
dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
validation_split: validation
fewshot_split: dev
output_type: generate_until
doc_to_text: "Q: {{question.strip()}}\n(A) {{choices[0]}} (B) {{choices[1]}} (C) {{choices[2]}} (D) {{choices[3]}}\nA: Let's think step by step."
doc_to_target: "{{['(A)', '(B)', '(C)', '(D)'][answer]}}"
filter_list:
- name: "get-answer"
filter:
- function: "regex"
regex_pattern: "((?<=The answer is )(.*)(?=.)|(?<=the answer is )(.*)(?=.)|(?<=The answer: )(.*)(?=.)|(?<=The final answer: )(.*)(?=.))"
- function: "take_first"
generation_kwargs:
until:
- "</s>"
do_sample: false
temperature: 0.0
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu_flan_generative_template_yaml deleted 100644 → 0
View file @ b1ba4e71
group: mmlu_flan_cot_zeroshot
dataset_path: cais/mmlu
validation_split: validation
fewshot_split: dev
output_type: greedy_until
doc_to_text: "Q: {{question.strip()}}\n(A) {{choices[0]}} (B) {{choices[1]}} (C) {{choices[2]}} (D) {{choices[3]}}\nA: Let's think step by step."
doc_to_target: "{{['(A)', '(B)', '(C)', '(D)'][answer]}}"
filter_list:
- name: "get-answer"
filter:
- function: "regex"
regex_pattern: "((?<=The answer is )(.*)(?=.)|(?<=the answer is )(.*)(?=.)|(?<=The answer: )(.*)(?=.)|(?<=The final answer: )(.*)(?=.))"
- function: "take_first"
generation_kwargs:
until:
- "</s>"
do_sample: false
temperature: 0.0
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_abstract_algebra.yaml
View file @ 2184b8de
dataset_name: abstract_algebra
description: 'The following are multiple choice questions (with answers) about abstract
algebra.
'
include: _mmlu_flan_generative_template_yaml
task: mmlu_flan_cot_zeroshot_abstract_algebra
"dataset_name": "abstract_algebra"
"description": "The following are multiple choice questions (with answers) about abstract\
\ algebra.\n\n"
"group": "mmlu_flan_cot_zeroshot_stem"
"include": "_mmlu_flan_cot_zeroshot_template_yaml"
"task": "mmlu_flan_cot_zeroshot_abstract_algebra"
lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_anatomy.yaml
View file @ 2184b8de
dataset_name: anatomy
description: 'The following are multiple choice questions (with answers) about anatomy.
'
include: _mmlu_flan_generative_template_yaml
task: mmlu_flan_cot_zeroshot_anatomy
"dataset_name": "anatomy"
"description": "The following are multiple choice questions (with answers) about anatomy.\n\
\n"
"group": "mmlu_flan_cot_zeroshot_stem"
"include": "_mmlu_flan_cot_zeroshot_template_yaml"
"task": "mmlu_flan_cot_zeroshot_anatomy"
lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_astronomy.yaml
View file @ 2184b8de
dataset_name: astronomy
description: 'The following are multiple choice questions (with answers) about astronomy.
'
include: _mmlu_flan_generative_template_yaml
task: mmlu_flan_cot_zeroshot_astronomy
"dataset_name": "astronomy"
"description": "The following are multiple choice questions (with answers) about astronomy.\n\
\n"
"group": "mmlu_flan_cot_zeroshot_stem"
"include": "_mmlu_flan_cot_zeroshot_template_yaml"
"task": "mmlu_flan_cot_zeroshot_astronomy"
lm_eval/tasks/mmlu/flan_cot_zeroshot/mmlu_business_ethics.yaml
View file @ 2184b8de
dataset_name: business_ethics
description: 'The following are multiple choice questions (with answers) about business
ethics.
'
include: _mmlu_flan_generative_template_yaml
task: mmlu_flan_cot_zeroshot_business_ethics
"dataset_name": "business_ethics"
"description": "The following are multiple choice questions (with answers) about business\
\ ethics.\n\n"
"group": "mmlu_flan_cot_zeroshot_other"
"include": "_mmlu_flan_cot_zeroshot_template_yaml"
"task": "mmlu_flan_cot_zeroshot_business_ethics"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment