Merge pull request #1089 from h-albert-lee/K-MMLU

Add kmmlu evaluation to tasks

lm_eval/tasks/kmmlu/README.md

+# k_mmlu
+
+### Paper
+
+Title: `K-MMLU(work in progress) `
+* ongoing project at publishing help/ non-english benchmark
+
+Abstract: `The K-MMLU (Korean-MMLU) is a comprehensive suite designed to evaluate the advanced knowledge and reasoning abilities of large language models (LLMs) within the Korean language and cultural context. This suite encompasses 45 topics, primarily focusing on expert-level subjects. It includes general subjects like Physics and Ecology, and law and political science, alongside specialized fields such as Non-Destructive Training and Maritime Engineering. The datasets are derived from Korean licensing exams, with about 90% of the questions including human accuracy based on the performance of human test-takers in these exams. K-MMLU is segmented into training, testing, and development subsets, with the test subset ranging from a minimum of 100 to a maximum of 1000 questions, totaling 35,000 questions. Additionally, a set of 10 questions is provided as a development set for few-shot exemplar development. At total, K-MMLU consists of 254,334 instances.`
+
+Homepage: https://huggingface.co/datasets/HAERAE-HUB/K-MMLU-Preview
+
+### Citation
+
+```
+We'll be updating this section soon.
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* `kmmlu`: 'All 45 subjects of the KMMLU dataset, evaluated following the methodology in MMLU's original implementation'
+
+#### Tasks
+
+The following tasks evaluate subjects in the KMMLU dataset
+- `kmmlu_{subject_english}`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

lm_eval/tasks/kmmlu/_default_kmmlu_yaml

+group: kmmlu
+dataset_path: HAERAE-HUB/K-MMLU-Preview
+output_type: multiple_choice
+training_split: train
+validation_split: dev
+test_split: test
+fewshot_split: dev
+output_type: multiple_choice
+process_docs: !function utils.process_docs
+doc_to_text: "{{question}}"
+doc_to_choice: "{{choices}}"
+doc_to_target: "{{gold}}"
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  - version: 0.0

lm_eval/tasks/kmmlu/kmmlu_accounting.yaml

+"dataset_name": "Accounting"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_accounting"

lm_eval/tasks/kmmlu/kmmlu_agricultural_sciences.yaml

+"dataset_name": "Agricultural Sciences"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_agricultural_sciences"

lm_eval/tasks/kmmlu/kmmlu_aviation_engineering_and_maintenance.yaml

+"dataset_name": "Aviation Engineering and Maintenance"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_aviation_engineering_and_maintenance"

lm_eval/tasks/kmmlu/kmmlu_biology.yaml

+"dataset_name": "Biology"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_biology"

lm_eval/tasks/kmmlu/kmmlu_chemical_engineering.yaml

+"dataset_name": "Chemical Engineering"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_chemical_engineering"

lm_eval/tasks/kmmlu/kmmlu_chemistry.yaml

+"dataset_name": "Chemistry"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_chemistry"

lm_eval/tasks/kmmlu/kmmlu_civil_engineering.yaml

+"dataset_name": "Civil Engineering"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_civil_engineering"

lm_eval/tasks/kmmlu/kmmlu_computer_science.yaml

+"dataset_name": "Computer Science"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_computer_science"

lm_eval/tasks/kmmlu/kmmlu_construction.yaml

+"dataset_name": "Construction"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_construction"

lm_eval/tasks/kmmlu/kmmlu_criminal_law.yaml

+"dataset_name": "Criminal Law"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_criminal_law"

lm_eval/tasks/kmmlu/kmmlu_ecology.yaml

+"dataset_name": "Ecology"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_ecology"

lm_eval/tasks/kmmlu/kmmlu_economics.yaml

+"dataset_name": "Economics"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_economics"

lm_eval/tasks/kmmlu/kmmlu_education.yaml

+"dataset_name": "Education"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_education"

lm_eval/tasks/kmmlu/kmmlu_electrical_engineering.yaml

+"dataset_name": "Electrical Engineering"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_electrical_engineering"

lm_eval/tasks/kmmlu/kmmlu_electronics_engineering.yaml

+"dataset_name": "Electronics Engineering"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_electronics_engineering"

lm_eval/tasks/kmmlu/kmmlu_energy_management.yaml

+"dataset_name": "Energy Management"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_energy_management"

lm_eval/tasks/kmmlu/kmmlu_environmental_science.yaml

+"dataset_name": "Environmental Science"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_environmental_science"

lm_eval/tasks/kmmlu/kmmlu_fashion.yaml

+"dataset_name": "Fashion"
+"include": "_default_kmmlu_yaml"
+"task": "kmmlu_fashion"