Skip to content

GitLab

Commit 13c6f5e7 authored by haileyschoelkopf's avatar haileyschoelkopf
Browse files

add draft cmmlu port

parent 9d0df41b
Hide whitespace changes
Inline Side-by-side
Showing with 386 additions and 1 deletion
.pre-commit-config.yaml
View file @ 13c6f5e7
......@@ -40,7 +40,7 @@ repos:
- id: codespell
exclude: >
(?x)^(
.*\.json|ignore.txt
.*\.json|ignore.txt|.*yaml
)$
args: [--check-filenames, --check-hidden, --ignore-words=ignore.txt]
- repo: https://github.com/pre-commit/mirrors-mypy
......
lm_eval/tasks/cmmlu/README.md 0 → 100644
View file @ 13c6f5e7
# CMMLU
### Paper
CMMLU: Measuring massive multitask language understanding in Chinese
https://arxiv.org/abs/2306.09212
CMMLU is a comprehensive evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of LLMs within the context of Chinese language and culture.
CMMLU covers a wide range of subjects, comprising 67 topics that span from elementary to advanced professional levels.
Homepage: https://github.com/haonan-li/CMMLU
### Citation
```bibtex
@misc{li2023cmmlu,
title={CMMLU: Measuring massive multitask language understanding in Chinese},
author={Haonan Li and Yixuan Zhang and Fajri Koto and Yifei Yang and Hai Zhao and Yeyun Gong and Nan Duan and Timothy Baldwin},
year={2023},
eprint={2306.09212},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
| Tasks |Version|Filter| Metric |Value | |Stderr|
|--------------------------------------------|-------|------|--------------------|-----:|---|-----:|
|cmmlu |N/A |none |acc |0.2480| | |
| | |none |acc(sample agg) |0.2494| | |
| | |none |acc_norm |0.2480| | |
| | |none |acc_norm(sample agg)|0.2494| | |
|-cmmlu_modern_chinese |Yaml |none |acc |0.2500|± |0.0404|
| | |none |acc_norm |0.2500|± |0.0404|
|-cmmlu_world_history |Yaml |none |acc |0.2484|± |0.0342|
| | |none |acc_norm |0.2484|± |0.0342|
|-cmmlu_college_education |Yaml |none |acc |0.2523|± |0.0422|
| | |none |acc_norm |0.2523|± |0.0422|
|-cmmlu_international_law |Yaml |none |acc |0.2486|± |0.0319|
| | |none |acc_norm |0.2486|± |0.0319|
|-cmmlu_philosophy |Yaml |none |acc |0.1905|± |0.0385|
| | |none |acc_norm |0.1905|± |0.0385|
|-cmmlu_professional_psychology |Yaml |none |acc |0.2457|± |0.0283|
| | |none |acc_norm |0.2457|± |0.0283|
|-cmmlu_college_engineering_hydrology |Yaml |none |acc |0.2830|± |0.0440|
| | |none |acc_norm |0.2830|± |0.0440|
|-cmmlu_electrical_engineering |Yaml |none |acc |0.2442|± |0.0329|
| | |none |acc_norm |0.2442|± |0.0329|
|-cmmlu_ancient_chinese |Yaml |none |acc |0.2378|± |0.0333|
| | |none |acc_norm |0.2378|± |0.0333|
|-cmmlu_chinese_food_culture |Yaml |none |acc |0.2353|± |0.0365|
| | |none |acc_norm |0.2353|± |0.0365|
|-cmmlu_chinese_literature |Yaml |none |acc |0.2598|± |0.0308|
| | |none |acc_norm |0.2598|± |0.0308|
|-cmmlu_legal_and_moral_basis |Yaml |none |acc |0.2477|± |0.0296|
| | |none |acc_norm |0.2477|± |0.0296|
|-cmmlu_construction_project_management |Yaml |none |acc |0.2374|± |0.0362|
| | |none |acc_norm |0.2374|± |0.0362|
|-cmmlu_ethnology |Yaml |none |acc |0.2519|± |0.0375|
| | |none |acc_norm |0.2519|± |0.0375|
|-cmmlu_high_school_geography |Yaml |none |acc |0.2542|± |0.0403|
| | |none |acc_norm |0.2542|± |0.0403|
|-cmmlu_professional_medicine |Yaml |none |acc |0.2500|± |0.0224|
| | |none |acc_norm |0.2500|± |0.0224|
|-cmmlu_global_facts |Yaml |none |acc |0.2349|± |0.0348|
| | |none |acc_norm |0.2349|± |0.0348|
|-cmmlu_astronomy |Yaml |none |acc |0.2303|± |0.0329|
| | |none |acc_norm |0.2303|± |0.0329|
|-cmmlu_machine_learning |Yaml |none |acc |0.2541|± |0.0396|
| | |none |acc_norm |0.2541|± |0.0396|
|-cmmlu_high_school_politics |Yaml |none |acc |0.2378|± |0.0357|
| | |none |acc_norm |0.2378|± |0.0357|
|-cmmlu_chinese_civil_service_exam |Yaml |none |acc |0.2562|± |0.0346|
| | |none |acc_norm |0.2562|± |0.0346|
|-cmmlu_professional_law |Yaml |none |acc |0.2512|± |0.0299|
| | |none |acc_norm |0.2512|± |0.0299|
|-cmmlu_college_medical_statistics |Yaml |none |acc |0.2453|± |0.0420|
| | |none |acc_norm |0.2453|± |0.0420|
|-cmmlu_computer_security |Yaml |none |acc |0.2573|± |0.0335|
| | |none |acc_norm |0.2573|± |0.0335|
|-cmmlu_food_science |Yaml |none |acc |0.2238|± |0.0350|
| | |none |acc_norm |0.2238|± |0.0350|
|-cmmlu_security_study |Yaml |none |acc |0.2519|± |0.0375|
| | |none |acc_norm |0.2519|± |0.0375|
|-cmmlu_high_school_physics |Yaml |none |acc |0.2545|± |0.0417|
| | |none |acc_norm |0.2545|± |0.0417|
|-cmmlu_management |Yaml |none |acc |0.2476|± |0.0299|
| | |none |acc_norm |0.2476|± |0.0299|
|-cmmlu_professional_accounting |Yaml |none |acc |0.2514|± |0.0329|
| | |none |acc_norm |0.2514|± |0.0329|
|-cmmlu_human_sexuality |Yaml |none |acc |0.2222|± |0.0372|
| | |none |acc_norm |0.2222|± |0.0372|
|-cmmlu_marxist_theory |Yaml |none |acc |0.2487|± |0.0315|
| | |none |acc_norm |0.2487|± |0.0315|
|-cmmlu_agronomy |Yaml |none |acc |0.2426|± |0.0331|
| | |none |acc_norm |0.2426|± |0.0331|
|-cmmlu_chinese_teacher_qualification |Yaml |none |acc |0.2626|± |0.0330|
| | |none |acc_norm |0.2626|± |0.0330|
|-cmmlu_genetics |Yaml |none |acc |0.2273|± |0.0317|
| | |none |acc_norm |0.2273|± |0.0317|
|-cmmlu_sports_science |Yaml |none |acc |0.2727|± |0.0348|
| | |none |acc_norm |0.2727|± |0.0348|
|-cmmlu_elementary_commonsense |Yaml |none |acc |0.2424|± |0.0305|
| | |none |acc_norm |0.2424|± |0.0305|
|-cmmlu_logical |Yaml |none |acc |0.1951|± |0.0359|
| | |none |acc_norm |0.1951|± |0.0359|
|-cmmlu_chinese_history |Yaml |none |acc |0.2508|± |0.0242|
| | |none |acc_norm |0.2508|± |0.0242|
|-cmmlu_traditional_chinese_medicine |Yaml |none |acc |0.2378|± |0.0314|
| | |none |acc_norm |0.2378|± |0.0314|
|-cmmlu_elementary_mathematics |Yaml |none |acc |0.2609|± |0.0290|
| | |none |acc_norm |0.2609|± |0.0290|
|-cmmlu_nutrition |Yaml |none |acc |0.2552|± |0.0363|
| | |none |acc_norm |0.2552|± |0.0363|
|-cmmlu_chinese_foreign_policy |Yaml |none |acc |0.1776|± |0.0371|
| | |none |acc_norm |0.1776|± |0.0371|
|-cmmlu_journalism |Yaml |none |acc |0.2616|± |0.0336|
| | |none |acc_norm |0.2616|± |0.0336|
|-cmmlu_jurisprudence |Yaml |none |acc |0.2506|± |0.0214|
| | |none |acc_norm |0.2506|± |0.0214|
|-cmmlu_sociology |Yaml |none |acc |0.2478|± |0.0288|
| | |none |acc_norm |0.2478|± |0.0288|
|-cmmlu_college_mathematics |Yaml |none |acc |0.2190|± |0.0406|
| | |none |acc_norm |0.2190|± |0.0406|
|-cmmlu_computer_science |Yaml |none |acc |0.2549|± |0.0306|
| | |none |acc_norm |0.2549|± |0.0306|
|-cmmlu_conceptual_physics |Yaml |none |acc |0.2517|± |0.0359|
| | |none |acc_norm |0.2517|± |0.0359|
|-cmmlu_elementary_chinese |Yaml |none |acc |0.2817|± |0.0284|
| | |none |acc_norm |0.2817|± |0.0284|
|-cmmlu_marketing |Yaml |none |acc |0.2500|± |0.0324|
| | |none |acc_norm |0.2500|± |0.0324|
|-cmmlu_high_school_chemistry |Yaml |none |acc |0.2576|± |0.0382|
| | |none |acc_norm |0.2576|± |0.0382|
|-cmmlu_college_law |Yaml |none |acc |0.2315|± |0.0408|
| | |none |acc_norm |0.2315|± |0.0408|
|-cmmlu_chinese_driving_rule |Yaml |none |acc |0.2595|± |0.0384|
| | |none |acc_norm |0.2595|± |0.0384|
|-cmmlu_clinical_knowledge |Yaml |none |acc |0.2532|± |0.0283|
| | |none |acc_norm |0.2532|± |0.0283|
|-cmmlu_education |Yaml |none |acc |0.2761|± |0.0351|
| | |none |acc_norm |0.2761|± |0.0351|
|-cmmlu_high_school_mathematics |Yaml |none |acc |0.2927|± |0.0356|
| | |none |acc_norm |0.2927|± |0.0356|
|-cmmlu_college_actuarial_science |Yaml |none |acc |0.2736|± |0.0435|
| | |none |acc_norm |0.2736|± |0.0435|
|-cmmlu_arts |Yaml |none |acc |0.2313|± |0.0334|
| | |none |acc_norm |0.2313|± |0.0334|
|-cmmlu_public_relations |Yaml |none |acc |0.2471|± |0.0328|
| | |none |acc_norm |0.2471|± |0.0328|
|-cmmlu_college_medicine |Yaml |none |acc |0.2418|± |0.0260|
| | |none |acc_norm |0.2418|± |0.0260|
|-cmmlu_economics |Yaml |none |acc |0.2453|± |0.0342|
| | |none |acc_norm |0.2453|± |0.0342|
|-cmmlu_elementary_information_and_technology|Yaml |none |acc |0.2731|± |0.0289|
| | |none |acc_norm |0.2731|± |0.0289|
|-cmmlu_anatomy |Yaml |none |acc |0.2432|± |0.0354|
| | |none |acc_norm |0.2432|± |0.0354|
|-cmmlu_world_religions |Yaml |none |acc |0.2875|± |0.0359|
| | |none |acc_norm |0.2875|± |0.0359|
|-cmmlu_virology |Yaml |none |acc |0.2485|± |0.0333|
| | |none |acc_norm |0.2485|± |0.0333|
|-cmmlu_high_school_biology |Yaml |none |acc |0.2485|± |0.0333|
| | |none |acc_norm |0.2485|± |0.0333|
|-cmmlu_business_ethics |Yaml |none |acc |0.2584|± |0.0304|
| | |none |acc_norm |0.2584|± |0.0304|
|Groups|Version|Filter| Metric |Value | |Stderr|
|------|-------|------|--------------------|-----:|---|------|
|cmmlu |N/A |none |acc |0.2480| | |
| | |none |acc(sample agg) |0.2494| | |
| | |none |acc_norm |0.2480| | |
| | |none |acc_norm(sample agg)|0.2494| | |
lm_eval/tasks/cmmlu/_default_template_yaml 0 → 100644
View file @ 13c6f5e7
group: cmmlu
dataset_path: haonan-li/cmmlu
test_split: test
fewshot_split: dev
fewshot_config:
sampler: first_n
output_type: multiple_choice
doc_to_text: "{{Question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\n答案："
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: "{{['A', 'B', 'C', 'D'].index(Answer)}}"
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
lm_eval/tasks/cmmlu/_generate_configs.py 0 → 100644
View file @ 13c6f5e7
"""
Take in a YAML, and output all other splits with this YAML
"""
import os
import yaml
import argparse
from tqdm import tqdm
from lm_eval.logger import eval_logger
SUBJECTS = {
"agronomy": "农学",
"anatomy": "解剖学",
"ancient_chinese": "古汉语",
"arts": "艺术学",
"astronomy": "天文学",
"business_ethics": "商业伦理",
"chinese_civil_service_exam": "中国公务员考试",
"chinese_driving_rule": "中国驾驶规则",
"chinese_food_culture": "中国饮食文化",
"chinese_foreign_policy": "中国外交政策",
"chinese_history": "中国历史",
"chinese_literature": "中国文学",
"chinese_teacher_qualification": "中国教师资格",
"clinical_knowledge": "临床知识",
"college_actuarial_science": "大学精算学",
"college_education": "大学教育学",
"college_engineering_hydrology": "大学工程水文学",
"college_law": "大学法律",
"college_mathematics": "大学数学",
"college_medical_statistics": "大学医学统计",
"college_medicine": "大学医学",
"computer_science": "计算机科学",
"computer_security": "计算机安全",
"conceptual_physics": "概念物理学",
"construction_project_management": "建设工程管理",
"economics": "经济学",
"education": "教育学",
"electrical_engineering": "电气工程",
"elementary_chinese": "小学语文",
"elementary_commonsense": "小学常识",
"elementary_information_and_technology": "小学信息技术",
"elementary_mathematics": "初等数学",
"ethnology": "民族学",
"food_science": "食品科学",
"genetics": "遗传学",
"global_facts": "全球事实",
"high_school_biology": "高中生物",
"high_school_chemistry": "高中化学",
"high_school_geography": "高中地理",
"high_school_mathematics": "高中数学",
"high_school_physics": "高中物理学",
"high_school_politics": "高中政治",
"human_sexuality": "人类性行为",
"international_law": "国际法学",
"journalism": "新闻学",
"jurisprudence": "法理学",
"legal_and_moral_basis": "法律与道德基础",
"logical": "逻辑学",
"machine_learning": "机器学习",
"management": "管理学",
"marketing": "市场营销",
"marxist_theory": "马克思主义理论",
"modern_chinese": "现代汉语",
"nutrition": "营养学",
"philosophy": "哲学",
"professional_accounting": "专业会计",
"professional_law": "专业法学",
"professional_medicine": "专业医学",
"professional_psychology": "专业心理学",
"public_relations": "公共关系",
"security_study": "安全研究",
"sociology": "社会学",
"sports_science": "体育学",
"traditional_chinese_medicine": "中医中药",
"virology": "病毒学",
"world_history": "世界历史",
"world_religions": "世界宗教",
}
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--base_yaml_path", required=True)
parser.add_argument("--save_prefix_path", default="cmmlu")
parser.add_argument("--cot_prompt_path", default=None)
parser.add_argument("--task_prefix", default="")
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
# get filename of base_yaml so we can `"include": ` it in our other YAMLs.
base_yaml_name = os.path.split(args.base_yaml_path)[-1]
with open(args.base_yaml_path) as f:
base_yaml = yaml.full_load(f)
if args.cot_prompt_path is not None:
import json
with open(args.cot_prompt_path) as f:
cot_file = json.load(f)
for subject_eng, subject_zh in tqdm(SUBJECTS.items()):
if args.cot_prompt_path is not None:
description = cot_file[subject_eng]
else:
description = f"以下是关于{subject_zh}的单项选择题，请直接给出正确答案的选项。\n\n"
yaml_dict = {
"include": base_yaml_name,
"task": f"cmmlu_{args.task_prefix}_{subject_eng}"
if args.task_prefix != ""
else f"cmmlu_{subject_eng}",
"dataset_name": subject_eng,
"description": description,
}
file_save_path = args.save_prefix_path + f"_{subject_eng}.yaml"
eval_logger.info(f"Saving yaml for subset {subject_eng} to {file_save_path}")
with open(file_save_path, "w") as yaml_file:
yaml.dump(
yaml_dict,
yaml_file,
width=float("inf"),
allow_unicode=True,
default_style='"',
)
lm_eval/tasks/cmmlu/cmmlu_default_agronomy.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "agronomy"
"description": "以下是关于农学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_agronomy"
lm_eval/tasks/cmmlu/cmmlu_default_anatomy.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "anatomy"
"description": "以下是关于解剖学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_anatomy"
lm_eval/tasks/cmmlu/cmmlu_default_ancient_chinese.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "ancient_chinese"
"description": "以下是关于古汉语的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_ancient_chinese"
lm_eval/tasks/cmmlu/cmmlu_default_arts.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "arts"
"description": "以下是关于艺术学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_arts"
lm_eval/tasks/cmmlu/cmmlu_default_astronomy.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "astronomy"
"description": "以下是关于天文学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_astronomy"
lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "business_ethics"
"description": "以下是关于商业伦理的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_business_ethics"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_civil_service_exam.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_civil_service_exam"
"description": "以下是关于中国公务员考试的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_civil_service_exam"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_driving_rule.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_driving_rule"
"description": "以下是关于中国驾驶规则的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_driving_rule"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_food_culture.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_food_culture"
"description": "以下是关于中国饮食文化的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_food_culture"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_foreign_policy"
"description": "以下是关于中国外交政策的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_foreign_policy"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_history.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_history"
"description": "以下是关于中国历史的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_history"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_literature.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_literature"
"description": "以下是关于中国文学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_literature"
lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "chinese_teacher_qualification"
"description": "以下是关于中国教师资格的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_chinese_teacher_qualification"
lm_eval/tasks/cmmlu/cmmlu_default_clinical_knowledge.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "clinical_knowledge"
"description": "以下是关于临床知识的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_clinical_knowledge"
lm_eval/tasks/cmmlu/cmmlu_default_college_actuarial_science.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "college_actuarial_science"
"description": "以下是关于大学精算学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_college_actuarial_science"
lm_eval/tasks/cmmlu/cmmlu_default_college_education.yaml 0 → 100644
View file @ 13c6f5e7
"dataset_name": "college_education"
"description": "以下是关于大学教育学的单项选择题，请直接给出正确答案的选项。\n\n"
"include": "_default_template_yaml"
"task": "cmmlu_college_education"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment