Skip to content

GitLab

Unverified Commit 4902aaaf authored by Clémentine Fourrier's avatar Clémentine Fourrier Committed by GitHub
Browse files

Making hardcoded few shots compatible with the chat template mechanism (#1895) 



* init test 1

* fix

* this format seems to be working - need to update all other tasks with the new format

* bbh with few shot format

* fix fewshot bbh

* add mmlu flan cot

* samples of cot

* kmmlu

* fix gsm8k

* update keys for mmlu

* minerva math

* bbh

* fix

* fix samples

* small fixes to templates

* last prompt format change

* fixing prompt

* fixed minerva math format

* rm accidental commited file

* added doc for few shot samples

* Update lm_eval/loggers/evaluation_tracker.py

* Update lm_eval/loggers/evaluation_tracker.py

* Update docs/new_task_guide.md
Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* added check in sampler per code review

* added the system from a function, plus an example in minerva math

* style

* Apply suggestions from code review
Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* fix unit tests 1

* forcing use of test split

---------
Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>
parent f4f59251
Hide whitespace changes
Inline Side-by-side
Showing with 360 additions and 266 deletions
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_public_relations.yaml
View file @ 4902aaaf
"dataset_name": "public_relations" dataset_name: public_relations
"description": "The following are multiple choice questions (with answers) about public\ description: The following are multiple choice questions (with answers) about public
\ relations.\n\nQ: Earth Hour was a campaign launched by which organization?\n(A)\ relations.
\ Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund\nA: Let's think step by\ fewshot_config:
\ step. We refer to Wikipedia articles on public relations for help. Earth Hour\ sampler: first_n
\ is a worldwide movement oragnized launched by the World Wildlife Fund. The answer\ samples:
\ is (D).\n\nQ: In issues management, what is the most proactive approach to addressing\ - question: 'Earth Hour was a campaign launched by which organization?
\ negative or misleading information posted online about your organization?\n(A)\
\ Buy domain names that could be used by opposition groups. (B) Post anonymous comments\ (A) Greenpeace (B) The UN (C) Oxfam (D) World Wildlife Fund'
\ on blogs to combat this information. (C) Prepare a news release that discredits\ target: Let's think step by step. We refer to Wikipedia articles on public relations
\ the inaccurate information. (D) Make policy changes to address complaints highlighted\ for help. Earth Hour is a worldwide movement oragnized launched by the World
\ on these sites.\nA: Let's think step by step. We refer to Wikipedia articles on\ Wildlife Fund. The answer is (D).
\ public relations for help. In issues management, the most proactive approach to\ - question: 'In issues management, what is the most proactive approach to addressing
\ addressing negative or misleading information posted online is to make policy\ negative or misleading information posted online about your organization?
\ changes to address complaints highlighted on those sites. The answer is (D).\n\
\nQ: At which stage in the planning process would a situation analysis be carried\ (A) Buy domain names that could be used by opposition groups. (B) Post anonymous
\ out?\n(A) Defining the program (B) Planning the program (C) Taking action and\ comments on blogs to combat this information. (C) Prepare a news release that
\ implementing ideas (D) Evaluation of the program\nA: Let's think step by step.\ discredits the inaccurate information. (D) Make policy changes to address complaints
\ We refer to Wikipedia articles on public relations for help. Situation analyses\ highlighted on these sites.'
\ are typically carried out during the planning process stage of defining the program.\ target: Let's think step by step. We refer to Wikipedia articles on public relations
\ The answer is (A).\n\nQ: Which of these statements is true of the Vatican in 2010\ for help. In issues management, the most proactive approach to addressing negative
\ at the time of the accusations of child abuse cover-ups?\n(A) There was a coordinated\ or misleading information posted online is to make policy changes to address
\ media response. (B) Consistent messages were communicated. (C) Criticisms were\ complaints highlighted on those sites. The answer is (D).
\ taken as attacks on the Catholic Church. (D) The credibility of the Vatican was\ - question: 'At which stage in the planning process would a situation analysis be carried
\ upheld.\nA: Let's think step by step. We refer to Wikipedia articles on public\ out?
\ relations for help. In 2010 when there were accusations of child abuse cover-ups,\
\ the Vatican took those criticisms as attacks on the Catholic Church. The answer\ (A) Defining the program (B) Planning the program (C) Taking action and implementing
\ is (C).\n\nQ: What should a public relations media practitioner do if she does\ ideas (D) Evaluation of the program'
\ not know the answer to a reporter's question?\n(A) Give the reporter other information\ target: Let's think step by step. We refer to Wikipedia articles on public relations
\ she is certain is correct. (B) Say that the information is 'off the record' and\ for help. Situation analyses are typically carried out during the planning process
\ will be disseminated later. (C) Say 'I don't know' and promise to provide the\ stage of defining the program. The answer is (A).
\ information later. (D) Say 'no comment,' rather than appear uninformed.\nA: Let's\ - question: 'Which of these statements is true of the Vatican in 2010 at the time of
\ think step by step. We refer to Wikipedia articles on public relations for help.\ the accusations of child abuse cover-ups?
\ If a public relations media practitioner does not know the answer to a reporter's\
\ question, they should say 'I don't know' and offer to provide the information\ (A) There was a coordinated media response. (B) Consistent messages were communicated.
\ later. The answer is (C).\n\n" (C) Criticisms were taken as attacks on the Catholic Church. (D) The credibility
"group": "mmlu_flan_cot_fewshot_social_sciences" of the Vatican was upheld.'
"include": "_mmlu_flan_cot_fewshot_template_yaml" target: Let's think step by step. We refer to Wikipedia articles on public relations
"task": "mmlu_flan_cot_fewshot_public_relations" for help. In 2010 when there were accusations of child abuse cover-ups, the
Vatican took those criticisms as attacks on the Catholic Church. The answer
is (C).
- question: 'What should a public relations media practitioner do if she does not know
the answer to a reporter''s question?
(A) Give the reporter other information she is certain is correct. (B) Say that
the information is ''off the record'' and will be disseminated later. (C) Say
''I don''t know'' and promise to provide the information later. (D) Say ''no
comment,'' rather than appear uninformed.'
target: 'Let''s think step by step. We refer to Wikipedia articles on public relations
for help. If a public relations media practitioner does not know the answer
to a reporter''s question, they should say ''I don''t know'' and offer to provide
the information later. The answer is (C).'
group: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_public_relations
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_security_studies.yaml
View file @ 4902aaaf
"dataset_name": "security_studies" dataset_name: security_studies
"description": "The following are multiple choice questions (with answers) about security\ description: The following are multiple choice questions (with answers) about security
\ studies.\n\nQ: What are the frameworks of analysis within which terrorism has\ studies.
\ been considered (as of 2020)?\n(A) Competition between larger nations has resulted\ fewshot_config:
\ in some countries actively supporting terrorist groups to undermine the strength\ sampler: first_n
\ of rival states. Terrorist networks are extended patronage clubs maintained and\ samples:
\ paid for by their donor states and are conceptualised as being like state actors,\ - question: 'What are the frameworks of analysis within which terrorism has been considered
\ to be dealt with using military force. (B) Globalization has enabled the internationalization\ (as of 2020)?
\ of terrorist activities by opening up their operational space, although coordination\
\ is still managed from a geographical base. This suggests that terrorist groups\ (A) Competition between larger nations has resulted in some countries actively
\ are nationally structured which means that terrorism cannot be considered in terms\ supporting terrorist groups to undermine the strength of rival states. Terrorist
\ of a war to be defeated militarily without having serious implications on the\ networks are extended patronage clubs maintained and paid for by their donor
\ indigenous population. (C) Terrorism can be viewed as a problem to be resolved\ states and are conceptualised as being like state actors, to be dealt with using
\ by military means (war on terrorism), by normal police techniques (terrorism as\ military force. (B) Globalization has enabled the internationalization of terrorist
\ crime), or as a medical problem with underlying causes and symptoms (terrorism\ activities by opening up their operational space, although coordination is still
\ as disease). (D) Terrorism is viewed as a criminal problem. The criminalization\ managed from a geographical base. This suggests that terrorist groups are nationally
\ of terrorism has two important implications. Firstly, it suggests that terrorism\ structured which means that terrorism cannot be considered in terms of a war
\ can be eradicated - terrorists can be caught and brought to trial by normal judicial\ to be defeated militarily without having serious implications on the indigenous
\ proceedings thereby removing the threat from society - and secondly, it suggests\ population. (C) Terrorism can be viewed as a problem to be resolved by military
\ that preventative crime techniques are applicable to prevent its development.\n\ means (war on terrorism), by normal police techniques (terrorism as crime),
A: Let's think step by step. We refer to Wikipedia articles on security studies\ or as a medical problem with underlying causes and symptoms (terrorism as disease).
\ for help. (A) is wrong because it is not competition between larger nations that\ (D) Terrorism is viewed as a criminal problem. The criminalization of terrorism
\ causes terrorism. \n(B) is wrong because globalization is not the cause of terrorism.\n\ has two important implications. Firstly, it suggests that terrorism can be eradicated
(C) is correct because the US undertook the war on terrorism. \n(D) is wrong because\ - terrorists can be caught and brought to trial by normal judicial proceedings
\ preventative crime techniques will likely not end terrorism. The answer is (C).\n\ thereby removing the threat from society - and secondly, it suggests that preventative
\nQ: Which of the following is the best lens through which to investigate the role\ crime techniques are applicable to prevent its development.'
\ of child soldiers?\n(A) Child soldiers are victims of combat that need re-education\ target: "Let's think step by step. We refer to Wikipedia articles on security\
\ and rehabilitation. (B) Children and their mothers are not active subjects in\ \ studies for help. (A) is wrong because it is not competition between larger\
\ warfare and are best considered as subjects in the private sphere. (C) Children\ \ nations that causes terrorism. \n(B) is wrong because globalization is not\
\ are most often innocent bystanders in war and are best used as signifiers of peace.\ \ the cause of terrorism.\n(C) is correct because the US undertook the war on\
\ (D) Children have political subjecthood that is missed when they are considered\ \ terrorism. \n(D) is wrong because preventative crime techniques will likely\
\ as passive victims of warfare.\nA: Let's think step by step. We refer to Wikipedia\ \ not end terrorism. The answer is (C)."
\ articles on security studies for help. Child soliders as a political topic can\ - question: 'Which of the following is the best lens through which to investigate the
\ be missed when they are considered passive victims of warfare. The answer is (D).\n\ role of child soldiers?
\nQ: How can we best describe the relationship between the state-centric approach\
\ and the concept of human security?\n(A) There are such wide divisions within the\ (A) Child soldiers are victims of combat that need re-education and rehabilitation.
\ human security framework regarding the nature of threats and referent objects\ (B) Children and their mothers are not active subjects in warfare and are best
\ that no widely applicable comparisons between state-centric approaches and human\ considered as subjects in the private sphere. (C) Children are most often innocent
\ security can be drawn. (B) By adopting the framework of human security, the limitations\ bystanders in war and are best used as signifiers of peace. (D) Children have
\ of the realist state-centric approach become evident. Whilst human security defines\ political subjecthood that is missed when they are considered as passive victims
\ the referent object as the person or population, state-centric approaches prioritise\ of warfare.'
\ the security of the state, de-prioritizing the pursuit of human security. (C)\ target: Let's think step by step. We refer to Wikipedia articles on security studies
\ The state-centric approach to security is a faction of human security, usually\ for help. Child soliders as a political topic can be missed when they are considered
\ defined within the broad school of human security. By being state-centric this\ passive victims of warfare. The answer is (D).
\ approach prioritises the individual as the referent object in security studies.\ - question: 'How can we best describe the relationship between the state-centric approach
\ (D) Both the state-centric and human-centric approaches to security are mutually\ and the concept of human security?
\ exclusive and offer a sufficient analytic framework with which to understand the\
\ international security system. It is therefore the role of security analysts to\ (A) There are such wide divisions within the human security framework regarding
\ determine which of these substantial concepts is correct, and which should be\ the nature of threats and referent objects that no widely applicable comparisons
\ discarded.\nA: Let's think step by step. We refer to Wikipedia articles on security\ between state-centric approaches and human security can be drawn. (B) By adopting
\ studies for help. Human security focuses on a person or population whereas state-centric\ the framework of human security, the limitations of the realist state-centric
\ approaches focus on the state while deprioritizing human security. The answer\ approach become evident. Whilst human security defines the referent object as
\ is (B).\n\nQ: In order to become securitized, a threat must be presented in which\ the person or population, state-centric approaches prioritise the security of
\ of these ways?\n(A) As an existential threat that requires immediate and extraordinary\ the state, de-prioritizing the pursuit of human security. (C) The state-centric
\ action, posing a threat to the survival of the state or to societal security.\ approach to security is a faction of human security, usually defined within
\ (B) As requiring immediate and extraordinary action by the state, threatening\ the broad school of human security. By being state-centric this approach prioritises
\ the survival of a referent object and therefore warranting the use of measures\ the individual as the referent object in security studies. (D) Both the state-centric
\ not normally employed in the political realm. (C) As an urgent threat to the survival\ and human-centric approaches to security are mutually exclusive and offer a
\ of the referent object, so serious that it legitimises the employment of extraordinary\ sufficient analytic framework with which to understand the international security
\ action in response. (D) As an urgent threat to the survival of the audience that\ system. It is therefore the role of security analysts to determine which of
\ requires extraordinary or emergency measures.\nA: Let's think step by step. We\ these substantial concepts is correct, and which should be discarded.'
\ refer to Wikipedia articles on security studies for help. To be securitized, a\ target: Let's think step by step. We refer to Wikipedia articles on security studies
\ threat must be an urgent threat to the survival of the referent object. The answer\ for help. Human security focuses on a person or population whereas state-centric
\ is (C).\n\nQ: What distinguishes coercive diplomacy from military force?\n(A)\ approaches focus on the state while deprioritizing human security. The answer
\ Compellence is another term for coercive diplomacy, but covering a narrower set\ is (B).
\ of criteria; compellence covers those threats aimed at initiating adversary action.\ - question: 'In order to become securitized, a threat must be presented in which of
\ A threat to coerce a state to give up part of its territory would count as coercive\ these ways?
\ diplomacy, as long as that threat proactively initiates action before reactive\
\ diplomacy is taken. (B) Coercive diplomacy constitutes the threats of limited\ (A) As an existential threat that requires immediate and extraordinary action,
\ force to induce adversary's incentive to comply with the coercer's demands. It\ posing a threat to the survival of the state or to societal security. (B) As
\ is an influence strategy that is intended to obtain compliance: the use of force\ requiring immediate and extraordinary action by the state, threatening the survival
\ to defeat an opponent first does not count. It leaves an element of choice with\ of a referent object and therefore warranting the use of measures not normally
\ the target to comply, or to continue. (C) Military force, or the threat of military\ employed in the political realm. (C) As an urgent threat to the survival of
\ force, utilises fear to achieve strategic objectives. Coercive diplomacy is differentiated\ the referent object, so serious that it legitimises the employment of extraordinary
\ from this approach, because it does not use fear as a tool for coercing an adversary.\ action in response. (D) As an urgent threat to the survival of the audience
\ (D) Coercive diplomacy is employed to use force but to limit its effects on the\ that requires extraordinary or emergency measures.'
\ international community. Coercive diplomacy is an aggressive strategy that is\ target: Let's think step by step. We refer to Wikipedia articles on security studies
\ intended to obtain compliance through defeat. It does not leave an element of\ for help. To be securitized, a threat must be an urgent threat to the survival
\ choice with the target, the target either being forced to comply or engage in\ of the referent object. The answer is (C).
\ conflict. It seeks to control by imposing compliance by removing any opportunity\ - question: 'What distinguishes coercive diplomacy from military force?
\ for negotiation or concession.\nA: Let's think step by step. We refer to Wikipedia\
\ articles on security studies for help. Coercive diplomacy uses the threat of force\ (A) Compellence is another term for coercive diplomacy, but covering a narrower
\ to induce the opponent to comply with demands. The answer is (B).\n\n" set of criteria; compellence covers those threats aimed at initiating adversary
"group": "mmlu_flan_cot_fewshot_social_sciences" action. A threat to coerce a state to give up part of its territory would count
"include": "_mmlu_flan_cot_fewshot_template_yaml" as coercive diplomacy, as long as that threat proactively initiates action before
"task": "mmlu_flan_cot_fewshot_security_studies" reactive diplomacy is taken. (B) Coercive diplomacy constitutes the threats
of limited force to induce adversary''s incentive to comply with the coercer''s
demands. It is an influence strategy that is intended to obtain compliance:
the use of force to defeat an opponent first does not count. It leaves an element
of choice with the target to comply, or to continue. (C) Military force, or
the threat of military force, utilises fear to achieve strategic objectives.
Coercive diplomacy is differentiated from this approach, because it does not
use fear as a tool for coercing an adversary. (D) Coercive diplomacy is employed
to use force but to limit its effects on the international community. Coercive
diplomacy is an aggressive strategy that is intended to obtain compliance through
defeat. It does not leave an element of choice with the target, the target either
being forced to comply or engage in conflict. It seeks to control by imposing
compliance by removing any opportunity for negotiation or concession.'
target: 'Let''s think step by step. We refer to Wikipedia articles on security
studies for help. Coercive diplomacy uses the threat of force to induce the
opponent to comply with demands. The answer is (B).'
group: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_security_studies
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_sociology.yaml
View file @ 4902aaaf
"dataset_name": "sociology" dataset_name: sociology
"description": "The following are multiple choice questions (with answers) about sociology.\n\ description: The following are multiple choice questions (with answers) about sociology.
\nQ: Which of the following is not a problem associated with official statistics\ fewshot_config:
\ on strike action?\n(A) most strikes go unnoticed by employers and the mass media\ sampler: first_n
\ (B) not all industrial disputes will be reported by the employer (C) the definition\ samples:
\ of strikes excludes those that involve fewer than ten workers or last less than\ - question: 'Which of the following is not a problem associated with official statistics
\ one day (D) it is hard to compare strikes that were measured in different ways\n\ on strike action?
A: Let's think step by step. We refer to Wikipedia articles on sociology for help.\
\ Official statistics on strike action can be problematic because not all industrial\ (A) most strikes go unnoticed by employers and the mass media (B) not all industrial
\ disputes will be reported by employers, the definition of strikes excludes those\ disputes will be reported by the employer (C) the definition of strikes excludes
\ that involves fewer than ten workers or last less than one day, and it is hard\ those that involve fewer than ten workers or last less than one day (D) it is
\ to compare strikes that were measured in different ways. Thus, (A) is not a problem\ hard to compare strikes that were measured in different ways'
\ associated with official statistics on strike action. The answer is (A).\n\nQ:\ target: Let's think step by step. We refer to Wikipedia articles on sociology
\ What does Berger (1963) describe as a metaphor for social reality?\n(A) a fairground\ for help. Official statistics on strike action can be problematic because not
\ ride (B) a circus (C) a puppet theatre (D) a ballet\nA: Let's think step by step.\ all industrial disputes will be reported by employers, the definition of strikes
\ We refer to Wikipedia articles on sociology for help. Berger describes social\ excludes those that involves fewer than ten workers or last less than one day,
\ reality using the metaphor of a puppet theatre. The answer is (C).\n\nQ: The term\ and it is hard to compare strikes that were measured in different ways. Thus,
\ 'hegemony' refers to:\n(A) the tendency for the working class not to realize their\ (A) is not a problem associated with official statistics on strike action. The
\ own interests (B) a dominant ideology that legitimates economic, political and\ answer is (A).
\ cultural power (C) a form of dual consciousness based on ideology and everyday\ - question: 'What does Berger (1963) describe as a metaphor for social reality?
\ experiences (D) a mode of payment given for outstanding topiary\nA: Let's think\
\ step by step. We refer to Wikipedia articles on sociology for help. Hegemony refers\ (A) a fairground ride (B) a circus (C) a puppet theatre (D) a ballet'
\ to a dominant ideology that legitimates economic, policital, and cultural power.\ target: Let's think step by step. We refer to Wikipedia articles on sociology
\ The answer is (B).\n\nQ: The shift from 'civil religion' to 'common religion'\ for help. Berger describes social reality using the metaphor of a puppet theatre.
\ means that:\n(A) the increasing bureaucracy of the state has made religion only\ The answer is (C).
\ a marginal part of our lives (B) despite the weakening of traditional authority,\ - question: 'The term ''hegemony'' refers to:
\ our everyday lives and 'common sense' remain shaped by religious beliefs and values\
\ (C) religious participation in collective worship may have declined, but people\ (A) the tendency for the working class not to realize their own interests (B)
\ still practise their faiths in private (D) people are much more likely to discuss\ a dominant ideology that legitimates economic, political and cultural power
\ their religious beliefs in public, informal settings\nA: Let's think step by step.\ (C) a form of dual consciousness based on ideology and everyday experiences
\ We refer to Wikipedia articles on sociology for help. The shift from civil religion\ (D) a mode of payment given for outstanding topiary'
\ to common religion means that despite the weakening of traditional authority,\ target: Let's think step by step. We refer to Wikipedia articles on sociology
\ our everyday lives and common sense remain shaped by religious beliefs and values.\ for help. Hegemony refers to a dominant ideology that legitimates economic,
\ The answer is (B).\n\nQ: Which of the following did the post-war welfare state\ policital, and cultural power. The answer is (B).
\ of 1948 not aim to provide:\n(A) free health care and education for all (B) a\ - question: 'The shift from ''civil religion'' to ''common religion'' means that:
\ minimum wage (C) full employment (D) universal welfare\nA: Let's think step by\
\ step. We refer to Wikipedia articles on sociology for help. The post-war welfare\ (A) the increasing bureaucracy of the state has made religion only a marginal
\ state of 1948 aimed to provide free healthcare and education, full employment,\ part of our lives (B) despite the weakening of traditional authority, our everyday
\ and universal welfare. But it did not aim to provide a minimum wage. The answer\ lives and ''common sense'' remain shaped by religious beliefs and values (C)
\ is (B).\n\n" religious participation in collective worship may have declined, but people
"group": "mmlu_flan_cot_fewshot_social_sciences" still practise their faiths in private (D) people are much more likely to discuss
"include": "_mmlu_flan_cot_fewshot_template_yaml" their religious beliefs in public, informal settings'
"task": "mmlu_flan_cot_fewshot_sociology" target: Let's think step by step. We refer to Wikipedia articles on sociology
for help. The shift from civil religion to common religion means that despite
the weakening of traditional authority, our everyday lives and common sense
remain shaped by religious beliefs and values. The answer is (B).
- question: 'Which of the following did the post-war welfare state of 1948 not aim
to provide:
(A) free health care and education for all (B) a minimum wage (C) full employment
(D) universal welfare'
target: 'Let''s think step by step. We refer to Wikipedia articles on sociology
for help. The post-war welfare state of 1948 aimed to provide free healthcare
and education, full employment, and universal welfare. But it did not aim to
provide a minimum wage. The answer is (B).'
group: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_sociology
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_us_foreign_policy.yaml
View file @ 4902aaaf
"dataset_name": "us_foreign_policy" dataset_name: us_foreign_policy
"description": "The following are multiple choice questions (with answers) about us\ description: The following are multiple choice questions (with answers) about us foreign
\ foreign policy.\n\nQ: How did Donald Trump attack globalization in the 2016 campaign?\n\ policy.
(A) Globalization had made men like him too rich (B) Globalization only benefited\ fewshot_config:
\ certain American states, such as New York (C) Liberal elites had encouraged globalization,\ sampler: first_n
\ while 'ordinary Americans' lost jobs because of it (D) Globalization encouraged\ samples:
\ damaging trade wars\nA: Let's think step by step. We refer to Wikipedia articles\ - question: 'How did Donald Trump attack globalization in the 2016 campaign?
\ on us foreign policy for help. Trump attacked globalization because he believed\
\ ordinary Americans lost jobs due to it, and so he wanted to blame liberals who\ (A) Globalization had made men like him too rich (B) Globalization only benefited
\ had encouraged it. The answer is (C).\n\nQ: How did NSC-68 change U.S. strategy?\n\ certain American states, such as New York (C) Liberal elites had encouraged
(A) It globalized containment. (B) It militarized containment. (C) It called for\ globalization, while ''ordinary Americans'' lost jobs because of it (D) Globalization
\ the development of the hydrogen bomb. (D) All of the above\nA: Let's think step\ encouraged damaging trade wars'
\ by step. We refer to Wikipedia articles on us foreign policy for help. NSC-68\ target: Let's think step by step. We refer to Wikipedia articles on us foreign
\ outlined a variety of courses of action, including globalization of containment,\ policy for help. Trump attacked globalization because he believed ordinary Americans
\ militarization of contaiment, and the development of the hydrogen bomb. The answer\ lost jobs due to it, and so he wanted to blame liberals who had encouraged it.
\ is (D).\n\nQ: How do Defensive Realism and Offensive Realism differ in their explanation\ The answer is (C).
\ of state behaviour?\n(A) Defensive realists place greater emphasis on the role\ - question: 'How did NSC-68 change U.S. strategy?
\ of international institutions (B) Defensive realists place less emphasis on geographical\
\ factors (C) Offensive realists give more priority to the national interest than\ (A) It globalized containment. (B) It militarized containment. (C) It called
\ Defensive realists. (D) Defensive realists believe states are security maximizers,\ for the development of the hydrogen bomb. (D) All of the above'
\ while Offensive realists believe states to be power maximizers\nA: Let's think\ target: Let's think step by step. We refer to Wikipedia articles on us foreign
\ step by step. We refer to Wikipedia articles on us foreign policy for help. While\ policy for help. NSC-68 outlined a variety of courses of action, including globalization
\ defensive realism advocates that states are security maximizers, offensive realists\ of containment, militarization of contaiment, and the development of the hydrogen
\ think of states as power maximizers. The answer is (D).\n\nQ: The realm of policy\ bomb. The answer is (D).
\ decisions concerned primarily with relations between the United States and the\ - question: 'How do Defensive Realism and Offensive Realism differ in their explanation
\ rest of the world is known as\n(A) terrorism policy. (B) economic policy. (C)\ of state behaviour?
\ foreign policy. (D) international policy.\nA: Let's think step by step. We refer\
\ to Wikipedia articles on us foreign policy for help. The topic of policy decisions\ (A) Defensive realists place greater emphasis on the role of international institutions
\ concerns with relations between the US and the rest of the world is known as foreign\ (B) Defensive realists place less emphasis on geographical factors (C) Offensive
\ policy. The answer is (C).\n\nQ: How did the 2008 financial crisis affect America's\ realists give more priority to the national interest than Defensive realists.
\ international reputation?\n(A) It damaged support for the US model of political\ (D) Defensive realists believe states are security maximizers, while Offensive
\ economy and capitalism (B) It created anger at the United States for exaggerating\ realists believe states to be power maximizers'
\ the crisis (C) It increased support for American global leadership under President\ target: Let's think step by step. We refer to Wikipedia articles on us foreign
\ Obama (D) It reduced global use of the US dollar\nA: Let's think step by step.\ policy for help. While defensive realism advocates that states are security
\ We refer to Wikipedia articles on us foreign policy for help. The 2008 financial\ maximizers, offensive realists think of states as power maximizers. The answer
\ crisis damanged the international reputation of the American model of political\ is (D).
\ economy and capitalism. The answer is (A).\n\n" - question: 'The realm of policy decisions concerned primarily with relations between
"group": "mmlu_flan_cot_fewshot_social_sciences" the United States and the rest of the world is known as
"include": "_mmlu_flan_cot_fewshot_template_yaml"
"task": "mmlu_flan_cot_fewshot_us_foreign_policy" (A) terrorism policy. (B) economic policy. (C) foreign policy. (D) international
policy.'
target: Let's think step by step. We refer to Wikipedia articles on us foreign
policy for help. The topic of policy decisions concerns with relations between
the US and the rest of the world is known as foreign policy. The answer is (C).
- question: 'How did the 2008 financial crisis affect America''s international reputation?
(A) It damaged support for the US model of political economy and capitalism
(B) It created anger at the United States for exaggerating the crisis (C) It
increased support for American global leadership under President Obama (D) It
reduced global use of the US dollar'
target: 'Let''s think step by step. We refer to Wikipedia articles on us foreign
policy for help. The 2008 financial crisis damanged the international reputation
of the American model of political economy and capitalism. The answer is (A).'
group: mmlu_flan_cot_fewshot_social_sciences
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_us_foreign_policy
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_virology.yaml
View file @ 4902aaaf
"dataset_name": "virology" dataset_name: virology
"description": "The following are multiple choice questions (with answers) about virology.\n\ description: The following are multiple choice questions (with answers) about virology.
\nQ: The median survival time to AIDS and death was established by following:\n\ fewshot_config:
(A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters\ sampler: first_n
\ (D) High-risk seronegatives\nA: Let's think step by step. We refer to Wikipedia\ samples:
\ articles on virology for help. The median survival time to AIDS and death was\ - question: 'The median survival time to AIDS and death was established by following:
\ established as a result of the development of seroconverters. The answer is (C).\n\
\nQ: Which of the following is a morphological characteristic of the paramyxoviruses.\n\ (A) Seroprevalent HIV-infected individuals (B) Seronegatives (C) Seroconverters
(A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate\ (D) High-risk seronegatives'
\ viruses (C) Icosahedral viruses with envelope (D) Very large viruses\nA: Let's\ target: Let's think step by step. We refer to Wikipedia articles on virology for
\ think step by step. We refer to Wikipedia articles on virology for help. Paramyxoviruses\ help. The median survival time to AIDS and death was established as a result
\ are fragile viruses often visualised with RNA spewing from the inside. The answer\ of the development of seroconverters. The answer is (C).
\ is (A).\n\nQ: The most important goal of a behavioral intervention is:\n(A) Change\ - question: 'Which of the following is a morphological characteristic of the paramyxoviruses.
\ in behavior (B) Comprehensive coverage (C) Effective use of behavioral theory\
\ (D) Sustained behavior change\nA: Let's think step by step. We refer to Wikipedia\ (A) Fragile viruses often visualised with RNA spewing from the inside (B) Elongate
\ articles on virology for help. The prim goal of a behavioral intervention is to\ viruses (C) Icosahedral viruses with envelope (D) Very large viruses'
\ cause sustained behavior change. The answer is (D).\n\nQ: A key factor facilitating\ target: Let's think step by step. We refer to Wikipedia articles on virology for
\ the application of nested case-control studies from the MACS was:\n(A) Data collection\ help. Paramyxoviruses are fragile viruses often visualised with RNA spewing
\ (B) Establishment of a repository of biologic specimens (C) Participant interest\ from the inside. The answer is (A).
\ (D) Administration of the questionnaire by staff\nA: Let's think step by step.\ - question: 'The most important goal of a behavioral intervention is:
\ We refer to Wikipedia articles on virology for help. The Multicenter AIDS Cohort\
\ Study's use of nested case-control studies was facilitated by the establishment\ (A) Change in behavior (B) Comprehensive coverage (C) Effective use of behavioral
\ of a repository of biologic specimens. The answer is (B).\n\nQ: Why are parvoviruses\ theory (D) Sustained behavior change'
\ a highly impactful parasite?\n(A) Because they have no nucleic acid (B) They require\ target: Let's think step by step. We refer to Wikipedia articles on virology for
\ a helper virus (C) Only replicate in dividing cells (D) Can integrate into host\ help. The prim goal of a behavioral intervention is to cause sustained behavior
\ chromosomes\nA: Let's think step by step. We refer to Wikipedia articles on virology\ change. The answer is (D).
\ for help. Paroviruses are highly impactful because they do not have nucleic acid.\ - question: 'A key factor facilitating the application of nested case-control studies
\ The answer is (A).\n\n" from the MACS was:
"group": "mmlu_flan_cot_fewshot_other"
"include": "_mmlu_flan_cot_fewshot_template_yaml" (A) Data collection (B) Establishment of a repository of biologic specimens
"task": "mmlu_flan_cot_fewshot_virology" (C) Participant interest (D) Administration of the questionnaire by staff'
target: Let's think step by step. We refer to Wikipedia articles on virology for
help. The Multicenter AIDS Cohort Study's use of nested case-control studies
was facilitated by the establishment of a repository of biologic specimens.
The answer is (B).
- question: 'Why are parvoviruses a highly impactful parasite?
(A) Because they have no nucleic acid (B) They require a helper virus (C) Only
replicate in dividing cells (D) Can integrate into host chromosomes'
target: 'Let''s think step by step. We refer to Wikipedia articles on virology
for help. Paroviruses are highly impactful because they do not have nucleic
acid. The answer is (A).'
group: mmlu_flan_cot_fewshot_other
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_virology
lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_world_religions.yaml
View file @ 4902aaaf
"dataset_name": "world_religions" dataset_name: world_religions
"description": "The following are multiple choice questions (with answers) about world\ description: The following are multiple choice questions (with answers) about world
\ religions.\n\nQ: How can the Upanishads be characterized?\n(A) Ritual texts (B)\ religions.
\ Philosophical texts (C) Hymns (D) Origin stories\nA: Let's think step by step.\ fewshot_config:
\ We refer to Wikipedia articles on world religions for help. The Upanishads are\ sampler: first_n
\ the most recent part of Vedas (the oldest scriptures in Hinduism) and supplied\ samples:
\ the basis of later Hindu philosophy. So they are philosophical texts. The answer\ - question: 'How can the Upanishads be characterized?
\ is (B).\n\nQ: What is the Second Gem in Buddhism?\n(A) The Dharma (B) The Sangha\
\ (C) The Buddha (D) The Bodhisattva\nA: Let's think step by step. We refer to Wikipedia\ (A) Ritual texts (B) Philosophical texts (C) Hymns (D) Origin stories'
\ articles on world religions for help. The Second Gem in Buddhism is The Dharma.\ target: Let's think step by step. We refer to Wikipedia articles on world religions
\ The answer is (A).\n\nQ: Which Japanese government promoted a kind of national\ for help. The Upanishads are the most recent part of Vedas (the oldest scriptures
\ cult based on the emperor and his associations with kami?\n(A) Honen (B) Tanaka\ in Hinduism) and supplied the basis of later Hindu philosophy. So they are philosophical
\ (C) Tokugawa (D) Meiji\nA: Let's think step by step. We refer to Wikipedia articles\ texts. The answer is (B).
\ on world religions for help. The promotion of a national cult based on the emperor\ - question: 'What is the Second Gem in Buddhism?
\ and his associations with Kami happened during the reign of Emperor Meiji (1852-1912).\
\ The answer is (D).\n\nQ: In which dynasty was the \"Mandate of Heaven\" developed\ (A) The Dharma (B) The Sangha (C) The Buddha (D) The Bodhisattva'
\ to legitimatize the new rulers?\n(A) Shang (B) Zhou (C) Han (D) Xia\nA: Let's\ target: Let's think step by step. We refer to Wikipedia articles on world religions
\ think step by step. We refer to Wikipedia articles on world religions for help.\ for help. The Second Gem in Buddhism is The Dharma. The answer is (A).
\ The \"Mandate of Heaven\" was developed as an ancient Chinese philosophical concept\ - question: 'Which Japanese government promoted a kind of national cult based on the
\ during the Zhou Dynasty (1046-256 BCE). The answer is (B).\n\nQ: What is the sign\ emperor and his associations with kami?
\ of the covenant for Jewish males?\n(A) The rainbow (B) Circumcision (C) A son\
\ (D) Bar mitzvah\nA: Let's think step by step. We refer to Wikipedia articles on\ (A) Honen (B) Tanaka (C) Tokugawa (D) Meiji'
\ world religions for help. In Judaism, the most distinctive sign of the covenant\ target: Let's think step by step. We refer to Wikipedia articles on world religions
\ is circumcision (brit milah). The answer is (B).\n\n" for help. The promotion of a national cult based on the emperor and his associations
"group": "mmlu_flan_cot_fewshot_humanities" with Kami happened during the reign of Emperor Meiji (1852-1912). The answer
"include": "_mmlu_flan_cot_fewshot_template_yaml" is (D).
"task": "mmlu_flan_cot_fewshot_world_religions" - question: 'In which dynasty was the "Mandate of Heaven" developed to legitimatize
the new rulers?
(A) Shang (B) Zhou (C) Han (D) Xia'
target: Let's think step by step. We refer to Wikipedia articles on world religions
for help. The "Mandate of Heaven" was developed as an ancient Chinese philosophical
concept during the Zhou Dynasty (1046-256 BCE). The answer is (B).
- question: 'What is the sign of the covenant for Jewish males?
(A) The rainbow (B) Circumcision (C) A son (D) Bar mitzvah'
target: 'Let''s think step by step. We refer to Wikipedia articles on world religions
for help. In Judaism, the most distinctive sign of the covenant is circumcision
(brit milah). The answer is (B).'
group: mmlu_flan_cot_fewshot_humanities
include: _mmlu_flan_cot_fewshot_template_yaml
task: mmlu_flan_cot_fewshot_world_religions
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment