Add Open Arabic LLM Leaderboard Benchmarks (Full and Light Version) (#2232)

* arabic leaferboard yaml file is added

* arabic toxigen is implemented

* Dataset library is imported

* arabic sciq is added

* util file of arabic toxigen is updated

* arabic race is added

* arabic piqa is implemented

* arabic open qa is added

* arabic copa is implemented

* arabic boolq ia added

* arabic arc easy is added

* arabic arc challenge is added

* arabic exams benchmark is implemented

* arabic hellaswag is added

* arabic leaderboard yaml file metrics are updated

* arabic mmlu benchmarks are added

* arabic mmlu group yaml file is updated

* alghafa benchmarks are added

* acva benchmarks are added

* acva utils.py is updated

* light version of arabic leaderboard benchmarks are added

* bugs fixed

* bug fixed

* bug fixed

* bug fixed

* bug fixed

* bug fixed

* library import bug is fixed

* doc to target updated

* bash file is deleted

* results folder is deleted

* leaderboard groups are added

* full arabic leaderboard groups are added, plus some bug fixes to the light version

* Create README.md

README.md for arabic_leaderboard_complete

* Create README.md

README.md for arabic_leaderboard_light

* Delete lm_eval/tasks/arabic_leaderboard directory

* Update README.md

* Update README.md

adding the Arabic leaderboards to the library

* Update README.md

10% of the training set

* Update README.md

10% of the training set

* revert .gitignore to prev version

* Update lm_eval/tasks/README.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* updated main README.md

* Update lm_eval/tasks/README.md

* specify machine translated benchmarks (complete)

* specify machine translated benchmarks (light version)

* add alghafa to the related task names (complete and light)

* add 'acva' to the related task names (complete and light)

* add 'arabic_leaderboard' to all the groups (complete and light)

* all dataset - not a random sample

* added more accurate details to the readme file

* added mt_mmlu from okapi

* Update lm_eval/tasks/README.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* Update lm_eval/tasks/README.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* updated mt_mmlu readme

* renaming 'alghafa' full and light

* renaming 'arabic_mmlu' light and full

* renaming 'acva' full and light

* update readme and standardize dir/file names

* running pre-commit

---------
Co-authored-by: shahrzads <sayehban@ualberta.ca>
Co-authored-by: shahrzads <56282669+shahrzads@users.noreply.github.com>
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_light.yaml

+group: arabic_leaderboard_arabic_mmlu_light
+task:
+  - arabic_leaderboard_arabic_mmlu_abstract_algebra_light
+  - arabic_leaderboard_arabic_mmlu_anatomy_light
+  - arabic_leaderboard_arabic_mmlu_astronomy_light
+  - arabic_leaderboard_arabic_mmlu_business_ethics_light
+  - arabic_leaderboard_arabic_mmlu_clinical_knowledge_light
+  - arabic_leaderboard_arabic_mmlu_college_biology_light
+  - arabic_leaderboard_arabic_mmlu_college_chemistry_light
+  - arabic_leaderboard_arabic_mmlu_college_computer_science_light
+  - arabic_leaderboard_arabic_mmlu_college_mathematics_light
+  - arabic_leaderboard_arabic_mmlu_college_medicine_light
+  - arabic_leaderboard_arabic_mmlu_college_physics_light
+  - arabic_leaderboard_arabic_mmlu_computer_security_light
+  - arabic_leaderboard_arabic_mmlu_conceptual_physics_light
+  - arabic_leaderboard_arabic_mmlu_econometrics_light
+  - arabic_leaderboard_arabic_mmlu_electrical_engineering_light
+  - arabic_leaderboard_arabic_mmlu_elementary_mathematics_light
+  - arabic_leaderboard_arabic_mmlu_formal_logic_light
+  - arabic_leaderboard_arabic_mmlu_global_facts_light
+  - arabic_leaderboard_arabic_mmlu_high_school_biology_light
+  - arabic_leaderboard_arabic_mmlu_high_school_chemistry_light
+  - arabic_leaderboard_arabic_mmlu_high_school_computer_science_light
+  - arabic_leaderboard_arabic_mmlu_high_school_european_history_light
+  - arabic_leaderboard_arabic_mmlu_high_school_geography_light
+  - arabic_leaderboard_arabic_mmlu_high_school_government_and_politics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_macroeconomics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_mathematics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_microeconomics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_physics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_psychology_light
+  - arabic_leaderboard_arabic_mmlu_high_school_statistics_light
+  - arabic_leaderboard_arabic_mmlu_high_school_us_history_light
+  - arabic_leaderboard_arabic_mmlu_high_school_world_history_light
+  - arabic_leaderboard_arabic_mmlu_human_aging_light
+  - arabic_leaderboard_arabic_mmlu_human_sexuality_light
+  - arabic_leaderboard_arabic_mmlu_international_law_light
+  - arabic_leaderboard_arabic_mmlu_jurisprudence_light
+  - arabic_leaderboard_arabic_mmlu_logical_fallacies_light
+  - arabic_leaderboard_arabic_mmlu_machine_learning_light
+  - arabic_leaderboard_arabic_mmlu_management_light
+  - arabic_leaderboard_arabic_mmlu_marketing_light
+  - arabic_leaderboard_arabic_mmlu_medical_genetics_light
+  - arabic_leaderboard_arabic_mmlu_miscellaneous_light
+  - arabic_leaderboard_arabic_mmlu_moral_disputes_light
+  - arabic_leaderboard_arabic_mmlu_moral_scenarios_light
+  - arabic_leaderboard_arabic_mmlu_nutrition_light
+  - arabic_leaderboard_arabic_mmlu_philosophy_light
+  - arabic_leaderboard_arabic_mmlu_prehistory_light
+  - arabic_leaderboard_arabic_mmlu_professional_accounting_light
+  - arabic_leaderboard_arabic_mmlu_professional_law_light
+  - arabic_leaderboard_arabic_mmlu_professional_medicine_light
+  - arabic_leaderboard_arabic_mmlu_professional_psychology_light
+  - arabic_leaderboard_arabic_mmlu_public_relations_light
+  - arabic_leaderboard_arabic_mmlu_security_studies_light
+  - arabic_leaderboard_arabic_mmlu_sociology_light
+  - arabic_leaderboard_arabic_mmlu_us_foreign_policy_light
+  - arabic_leaderboard_arabic_mmlu_virology_light
+  - arabic_leaderboard_arabic_mmlu_world_religions_light
+aggregate_metric_list:
+  - metric: acc
+    aggregation: mean
+    weight_by_size: true
+  - metric: acc_norm
+    aggregation: mean
+    weight_by_size: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_logical_fallacies_light.yaml

+task: arabic_leaderboard_arabic_mmlu_logical_fallacies_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: logical_fallacies
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_machine_learning_light.yaml

+task: arabic_leaderboard_arabic_mmlu_machine_learning_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: machine_learning
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_management_light.yaml

+task: arabic_leaderboard_arabic_mmlu_management_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: management
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_marketing_light.yaml

+task: arabic_leaderboard_arabic_mmlu_marketing_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: marketing
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_medical_genetics_light.yaml

+task: arabic_leaderboard_arabic_mmlu_medical_genetics_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: medical_genetics
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_miscellaneous_light.yaml

+task: arabic_leaderboard_arabic_mmlu_miscellaneous_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: miscellaneous
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_moral_disputes_light.yaml

+task: arabic_leaderboard_arabic_mmlu_moral_disputes_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: moral_disputes
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_moral_scenarios_light.yaml

+task: arabic_leaderboard_arabic_mmlu_moral_scenarios_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: moral_scenarios
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_nutrition_light.yaml

+task: arabic_leaderboard_arabic_mmlu_nutrition_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: nutrition
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_philosophy_light.yaml

+task: arabic_leaderboard_arabic_mmlu_philosophy_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: philosophy
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_prehistory_light.yaml

+task: arabic_leaderboard_arabic_mmlu_prehistory_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: prehistory
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_professional_accounting_light.yaml

+task: arabic_leaderboard_arabic_mmlu_professional_accounting_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: professional_accounting
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_professional_law_light.yaml

+task: arabic_leaderboard_arabic_mmlu_professional_law_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: professional_law
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_professional_medicine_light.yaml

+task: arabic_leaderboard_arabic_mmlu_professional_medicine_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: professional_medicine
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_professional_psychology_light.yaml

+task: arabic_leaderboard_arabic_mmlu_professional_psychology_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: professional_psychology
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_public_relations_light.yaml

+task: arabic_leaderboard_arabic_mmlu_public_relations_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: public_relations
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_security_studies_light.yaml

+task: arabic_leaderboard_arabic_mmlu_security_studies_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: security_studies
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_sociology_light.yaml

+task: arabic_leaderboard_arabic_mmlu_sociology_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: sociology
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm_eval/tasks/arabic_leaderboard_light/arabic_leaderboard_arabic_mmlu_light/arabic_leaderboard_arabic_mmlu_us_foreign_policy_light.yaml

+task: arabic_leaderboard_arabic_mmlu_us_foreign_policy_light
+dataset_path: arcee-globe/Arabic_MMLU-10percent
+dataset_name: us_foreign_policy
+output_type: multiple_choice
+training_split: null
+validation_split: dev
+test_split: test
+process_docs: !function utils.process_docs
+doc_to_text: "{{query}}"
+doc_to_target: "{{gold}}"
+doc_to_choice: "choices"
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0