Merge branch 'upstream' into 'mmlu-pro'

add tokenizer logs info (#1731)

See merge request shijie.yu/lm-evaluation-harness!4

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml

@@ -56,6 +56,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on government
       and politics for help. The US Constitution is not very specific about the powers
       of the president, leading to uncertainty over its limits. The answer is (A).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_government_and_politics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml

@@ -48,6 +48,6 @@ fewshot_config:
       for help. The economic transactions related to the performance of the American
       pop-singer in Paris happens entirely outside the U.S. and hence is not included
       in the GDP numbers. The answer is (C).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_macroeconomics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml

@@ -46,6 +46,6 @@ fewshot_config:
     target: 'Let''s think step by step. The least common multiple of 2, 3 and 5 is
       30, so during a 7 minute dance, all the three lights will come on at the same
       time $2*7+1=15$ times. The answer is (B).'
-group: mmlu_flan_cot_fewshot_stem
+tag: mmlu_flan_cot_fewshot_stem
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_mathematics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml

@@ -51,6 +51,6 @@ fewshot_config:
       for help. An increase in the construction of new houses means an increase demand
       of in-house painting, thus increases the demand for housepainters. The answer
       is (C).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_microeconomics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml

@@ -45,6 +45,6 @@ fewshot_config:
       cannot have any net displacement because the pipe closure stops them. So the
       particle displacement is at a node. This closure also causes the pressure to
       be maximal, i.e. an antinode. The answer is (B).'
-group: mmlu_flan_cot_fewshot_stem
+tag: mmlu_flan_cot_fewshot_stem
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_physics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml

@@ -59,6 +59,6 @@ fewshot_config:
       for help. People with an external locus of control believes fate and luck play
       an important role in their lives, while people with an internal locus of control
       believes they control their lives. The answer is (D).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_psychology

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml

@@ -76,6 +76,6 @@ fewshot_config:
       not perfectly correlated. Statement B is false because uncorrelated variables
       regression lines can have slope zero. Statement C is false because correlation
       is symmetric in the two random variables. The answer is (D).'
-group: mmlu_flan_cot_fewshot_stem
+tag: mmlu_flan_cot_fewshot_stem
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_statistics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml

@@ -151,6 +151,6 @@ fewshot_config:
       suspect Washington''s military response to Whiskey Rebellion. Bacon''s Rebellion
       and Pontiac''s Rebellion happen before the Revolution and they can be ruled
       out. The answer is (C).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_us_history

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml

@@ -95,6 +95,6 @@ fewshot_config:
       for help. Brahman refers to the ultimate reality of all things in the Hindu
       religion. In contrast, Buddhism does not have a concept of supreme God. The
       answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_high_school_world_history

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_aging.yaml

@@ -37,6 +37,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on human aging
       for help. Texas does not have state tax, and has low cost of living compared
       with the other three options. The answer is (A).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_human_aging

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_human_sexuality.yaml

@@ -45,6 +45,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on human sexuality
       for help. Morning sickness usually begins by nine weeks after conception, corresponding
       to the first trimester. The answer is (A).'
-group: mmlu_flan_cot_fewshot_social_sciences
+tag: mmlu_flan_cot_fewshot_social_sciences
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_human_sexuality

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_international_law.yaml

@@ -65,6 +65,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on international
       law for help. Article 2(4) of the UN Charter prohibits states from using armed
       forces in their international relations. The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_international_law

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_jurisprudence.yaml

@@ -54,6 +54,6 @@ fewshot_config:
       principle'', and reject the ''system of natural liberty'', but the POP would
       not choose equality above liberty, since the POP assume both equal and free
       citizens. The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_jurisprudence

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_logical_fallacies.yaml

@@ -56,6 +56,6 @@ fewshot_config:
       fallacies for help. The argument against the person fallacy occurs when someone
       irrelevantly attacks the character of an opposing arguer, instead of addressing
       that opponent''s arguments. The answer is (C).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_logical_fallacies

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_machine_learning.yaml

@@ -69,6 +69,6 @@ fewshot_config:
       \ epsilon when we have N samples if 2 exp(-2 epsilon^2 N)<0.05, this implies\
       \ that N > -1/(2*epsilon**2) log ( 0.05/2 )= log (40)*5000. Since log(40)>1,\
       \ we have that one needs more than 1000 examples. The answer is (D).\n\n"
-group: mmlu_flan_cot_fewshot_stem
+tag: mmlu_flan_cot_fewshot_stem
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_machine_learning

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_management.yaml

@@ -41,6 +41,6 @@ fewshot_config:
       for help. The Ohio State Leadership Studies conducted in the 1940s identified
       initiating structure and consideration as the two main dimensions of leader
       behavior. The answer is (D).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_management

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_marketing.yaml

@@ -51,6 +51,6 @@ fewshot_config:
     target: 'Let''s think step by step. We refer to Wikipedia articles on marketing
       for help. Geodemographics is a natural outcome when combining demographic and
       geographic variables. The answer is (A).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_marketing

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_medical_genetics.yaml

@@ -46,6 +46,6 @@ fewshot_config:
       genetics for help. A Southern blot is a method in molecular biology for detecting
       specific DNA sequences in a sample. Large triplet repeat expansions are usually
       detected with this method. The answer is (C).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_medical_genetics

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_miscellaneous.yaml

@@ -38,6 +38,6 @@ fewshot_config:
       (A) one (B) two (C) four (D) eight'
     target: 'Let''s think step by step. We refer to Wikipedia for help. Most cars
       have two axles to rotate the wheels.. The answer is (B).'
-group: mmlu_flan_cot_fewshot_other
+tag: mmlu_flan_cot_fewshot_other
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_miscellaneous

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_moral_disputes.yaml

@@ -59,6 +59,6 @@ fewshot_config:
       as it treats individuals as incapable of communal relations. It is unclear that
       capital punishment is to the benefit of, or a deterrent of harm to the community.
       The answer is (A).'
-group: mmlu_flan_cot_fewshot_humanities
+tag: mmlu_flan_cot_fewshot_humanities
 include: _mmlu_flan_cot_fewshot_template_yaml
 task: mmlu_flan_cot_fewshot_moral_disputes