Merge branch 'group-agg-rework' of...

Merge branch 'group-agg-rework' of https://github.com/EleutherAI/lm-evaluation-harness into multiprompt

lm_eval/tasks/agieval/sat-math.yaml

 include: aqua-rat.yaml
-group:
-  - agieval
-  - agieval_nous
-  - agieval_en
 task: agieval_sat_math
 dataset_path: hails/agieval-sat-math

lm_eval/tasks/ammlu/README.md → lm_eval/tasks/alghafa/copa_ar/README.md

-# ArabicMMLU
+#Arabic COPA
 
 ### Paper
 
-ArabicMMLU: Measuring massive multitask language understanding in Arabic
-This dataset has been translated from the original MMLU with the help of GPT-4.
+Original Title: `COPA`
 
-The original data [MMLU](https://arxiv.org/pdf/2009.03300v3.pdf)
 
-The translation has been done with AceGPT researchers [AceGPT](https://arxiv.org/abs/2309.12053)
 
-ArabicMMLU is a comprehensive evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of LLMs within the context of Arabic language and culture.
-ArabicMMLU covers a wide range of subjects, comprising 57 topics that span from elementary to advanced professional levels.
+The Choice Of Plausible Alternatives (COPA) evaluation provides researchers with a tool for assessing progress in open-domain commonsense causal reasoning.
 
-Homepage: [AceGPT Homepage](https://github.com/FreedomIntelligence/AceGPT/tree/main/eval/benchmark_eval/benchmarks/MMLUArabic)
+[Homepage](https://people.ict.usc.edu/~gordon/copa.html)
 
-### Citation
+AlGhafa has translated this dataset to Arabic[AlGafa](https://aclanthology.org/2023.arabicnlp-1.21.pdf)
+
+The link to the Arabic version of the dataset [PICA](https://gitlab.com/tiiuae/alghafa/-/tree/main/arabic-eval/copa_ar)
 
+### Citation
 
 ### Groups and Tasks
 
 #### Groups
 
-- `ammlu`: All 57 subjects of the ArabicMMLU dataset, evaluated following the methodology in MMLU's original implementation.
+* Not part of a group yet.
 
 #### Tasks
 
-
-The following tasks evaluate subjects in the ArabicMMLU dataset using loglikelihood-based multiple-choice scoring:
-- `ammlu_{subject_english}`
+* `copa_ar`
 
 ### Checklist
 
+For adding novel benchmarks/datasets to the library:
 * [x] Is the task an existing benchmark in the literature?
   * [x] Have you referenced the original paper that introduced the task?
-  * [x] If yes, does the original paper provide a reference implementation?
-    * [x] Yes, original implementation contributed by author of the benchmark
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
 
 If other tasks on this dataset are already supported:
 * [x] Is the "Main" variant of this task clearly denoted?

lm_eval/tasks/ammlu/_default_template_yaml → lm_eval/tasks/alghafa/copa_ar/copa_ar.yaml

-group: ammlu
-dataset_path: Hennara/ammlu
-test_split: test
-fewshot_split: dev
-fewshot_config:
-  sampler: first_n
+task: copa_ar
+dataset_path: Hennara/copa_ar
+dataset_name: null
 output_type: multiple_choice
-doc_to_text: "{{Question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nالجواب："
-doc_to_choice: ["A", "B", "C", "D"]
-doc_to_target: "{{['A', 'B', 'C', 'D'].index(Answer)}}"
+training_split: null
+validation_split: null
+test_split: test
+doc_to_text: "السؤال: {{query}}\nالجواب:"
+doc_to_choice: "{{[sol1, sol2]}}"
+doc_to_target: label
+should_decontaminate: true
+doc_to_decontamination_query: query
 metric_list:
   - metric: acc
     aggregation: mean
@@ -16,4 +18,4 @@ metric_list:
     aggregation: mean
     higher_is_better: true
 metadata:
-  version: 0.0
+  version: 1.0

lm_eval/tasks/alghafa/piqa_ar/README.md

+#Arabic PIQA
+
+### Paper
+
+Original Title: `PIQA: Reasoning about Physical Commonsense in Natural Language`
+
+Original paper: [PICA](https://arxiv.org/abs/1911.11641)
+
+Physical Interaction: Question Answering (PIQA) is a physical commonsense
+reasoning and a corresponding benchmark dataset. PIQA was designed to investigate
+the physical knowledge of existing models. To what extent are current approaches
+actually learning about the world?
+
+[Homepage](https://yonatanbisk.com/piqa)
+
+AlGhafa has translated this dataset to Arabic[AlGafa](https://aclanthology.org/2023.arabicnlp-1.21.pdf)
+
+The link to the Arabic version of the dataset [PICA](https://gitlab.com/tiiuae/alghafa/-/tree/main/arabic-eval/pica_ar)
+
+### Citation
+
+### Groups and Tasks
+
+#### Groups
+
+* Not part of a group yet.
+
+#### Tasks
+
+* `piqa_ar`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [x] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [x] Have you noted which, if any, published evaluation setups are matched by this variant?

lm_eval/tasks/alghafa/piqa_ar/piqa_ar.yaml

+task: piqa_ar
+dataset_path: Hennara/pica_ar
+dataset_name: null
+output_type: multiple_choice
+training_split: null
+validation_split: null
+test_split: test
+doc_to_text: "السؤال: {{goal}}\nالجواب:"
+doc_to_choice: "{{[sol1, sol2]}}"
+doc_to_target: label
+should_decontaminate: true
+doc_to_decontamination_query: goal
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/ammlu/_generate_configs.py

-"""
-Take in a YAML, and output all other splits with this YAML
-"""
-import argparse
-import os
-
-import yaml
-from tqdm import tqdm
-
-
-SUBJECTS = {
-    "abstract_algebra": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "anatomy": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "astronomy": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "business_ethics": "علوم أخرى",
-    "clinical_knowledge": "علوم أخرى",
-    "college_biology": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "college_chemistry": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "college_computer_science": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "college_mathematics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "college_medicine": "علوم أخرى",
-    "college_physics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "computer_security": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "conceptual_physics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "econometrics": "العلوم الإجتماعية",
-    "electrical_engineering": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "elementary_mathematics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "formal_logic": "العلوم الانسانية",
-    "global_facts": "علوم أخرى",
-    "high_school_biology": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_chemistry": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_computer_science": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_european_history": "العلوم الانسانية",
-    "high_school_geography": "العلوم الإجتماعية",
-    "high_school_government_and_politics": "العلوم الإجتماعية",
-    "high_school_macroeconomics": "العلوم الإجتماعية",
-    "high_school_mathematics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_microeconomics": "العلوم الإجتماعية",
-    "high_school_physics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_psychology": "العلوم الإجتماعية",
-    "high_school_statistics": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "high_school_us_history": "العلوم الانسانية",
-    "high_school_world_history": "العلوم الانسانية",
-    "human_aging": "علوم أخرى",
-    "human_sexuality": "العلوم الإجتماعية",
-    "international_law": "العلوم الانسانية",
-    "jurisprudence": "العلوم الانسانية",
-    "logical_fallacies": "العلوم الانسانية",
-    "machine_learning": "ألعلوم وتقنية المعلومات و الرياضيات",
-    "management": "علوم أخرى",
-    "marketing": "علوم أخرى",
-    "medical_genetics": "علوم أخرى",
-    "miscellaneous": "علوم أخرى",
-    "moral_disputes": "العلوم الانسانية",
-    "moral_scenarios": "العلوم الانسانية",
-    "nutrition": "علوم أخرى",
-    "philosophy": "العلوم الانسانية",
-    "prehistory": "العلوم الانسانية",
-    "professional_accounting": "علوم أخرى",
-    "professional_law": "العلوم الانسانية",
-    "professional_medicine": "علوم أخرى",
-    "professional_psychology": "العلوم الإجتماعية",
-    "public_relations": "العلوم الإجتماعية",
-    "security_studies": "العلوم الإجتماعية",
-    "sociology": "العلوم الإجتماعية",
-    "us_foreign_policy": "العلوم الإجتماعية",
-    "virology": "علوم أخرى",
-    "world_religions": "العلوم الانسانية",
-}
-
-
-def parse_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--base_yaml_path", required=True)
-    parser.add_argument("--save_prefix_path", default="ammlu")
-    parser.add_argument("--cot_prompt_path", default=None)
-    parser.add_argument("--task_prefix", default="")
-    return parser.parse_args()
-
-
-if __name__ == "__main__":
-    args = parse_args()
-
-    # get filename of base_yaml so we can `"include": ` it in our other YAMLs.
-    base_yaml_name = os.path.split(args.base_yaml_path)[-1]
-    with open(args.base_yaml_path, encoding="utf-8") as f:
-        base_yaml = yaml.full_load(f)
-
-    if args.cot_prompt_path is not None:
-        import json
-
-        with open(args.cot_prompt_path, encoding="utf-8") as f:
-            cot_file = json.load(f)
-
-    for subject_eng, category in tqdm(SUBJECTS.items()):
-        if args.cot_prompt_path is not None:
-            description = cot_file[subject_eng]
-        else:
-            description = f"فم بعملية التقييم في مجال {category} \n\n"
-
-        yaml_dict = {
-            "include": base_yaml_name,
-            "task": f"ammlu_{args.task_prefix}_{subject_eng}"
-            if args.task_prefix != ""
-            else f"ammlu_{subject_eng}",
-            "dataset_name": subject_eng,
-            "description": description,
-        }
-
-        file_save_path = args.save_prefix_path + f"_{subject_eng}.yaml"
-        print(f"Saving yaml for subset {subject_eng} to {file_save_path}")
-        with open(file_save_path, "w", encoding="utf-8") as yaml_file:
-            yaml.dump(
-                yaml_dict,
-                yaml_file,
-                width=float("inf"),
-                allow_unicode=True,
-                default_style='"',
-            )

lm_eval/tasks/ammlu/ammlu_abstract_algebra.yaml

-"dataset_name": "abstract_algebra"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_abstract_algebra"

lm_eval/tasks/ammlu/ammlu_anatomy.yaml

-"dataset_name": "anatomy"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_anatomy"

lm_eval/tasks/ammlu/ammlu_astronomy.yaml

-"dataset_name": "astronomy"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_astronomy"

lm_eval/tasks/ammlu/ammlu_college_biology.yaml

-"dataset_name": "college_biology"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_college_biology"

lm_eval/tasks/ammlu/ammlu_college_chemistry.yaml

-"dataset_name": "college_chemistry"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_college_chemistry"

lm_eval/tasks/ammlu/ammlu_college_computer_science.yaml

-"dataset_name": "college_computer_science"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_college_computer_science"

lm_eval/tasks/ammlu/ammlu_college_physics.yaml

-"dataset_name": "college_physics"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_college_physics"

lm_eval/tasks/ammlu/ammlu_computer_security.yaml

-"dataset_name": "computer_security"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_computer_security"

lm_eval/tasks/ammlu/ammlu_conceptual_physics.yaml

-"dataset_name": "conceptual_physics"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_conceptual_physics"

lm_eval/tasks/ammlu/ammlu_econometrics.yaml

-"dataset_name": "econometrics"
-"description": "فم بعملية التقييم في مجال العلوم الإجتماعية \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_econometrics"

lm_eval/tasks/ammlu/ammlu_formal_logic.yaml

-"dataset_name": "formal_logic"
-"description": "فم بعملية التقييم في مجال العلوم الانسانية \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_formal_logic"

lm_eval/tasks/ammlu/ammlu_high_school_computer_science.yaml

-"dataset_name": "high_school_computer_science"
-"description": "فم بعملية التقييم في مجال ألعلوم وتقنية المعلومات و الرياضيات \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_high_school_computer_science"

lm_eval/tasks/ammlu/ammlu_high_school_european_history.yaml

-"dataset_name": "high_school_european_history"
-"description": "فم بعملية التقييم في مجال العلوم الانسانية \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_high_school_european_history"

lm_eval/tasks/ammlu/ammlu_high_school_government_and_politics.yaml

-"dataset_name": "high_school_government_and_politics"
-"description": "فم بعملية التقييم في مجال العلوم الإجتماعية \n\n"
-"include": "_default_template_yaml"
-"task": "ammlu_high_school_government_and_politics"