Unverified Commit 9ae96cdf authored by ZoneTwelve's avatar ZoneTwelve Committed by GitHub
Browse files

TMMLU+ implementation (#1394)



* implementation of TMMLU+

* implemented: TMMLU+

****TMMLU+ : large-scale Traditional chinese Massive Multitask language Understanding****

- 4 categories
    - STEM
    - Social Science
    - Humanities
    - Other

The TMMLU+ dataset, encompassing over 67 subjects and 20160 tasks, is six times larger and more balanced than its predecessor, TMMLU, and includes benchmark results from both closed-source and 20 open-weight Chinese large language models with 1.8B to 72B parameters. However, Traditional Chinese variants continue to underperform compared to major Simplified Chinese models.

```markdown
Total number of tasks in the 'test' sets: 20160
Total number of tasks in the 'validation' sets: 2247
Total number of tasks in the 'train' sets: 335
```

* Remove print from __init__.py

There was my mistake in forgetting to remove the debug print from the code.

* update: move TMMLU+ config generation program into default

* fix: we should use training set as few shots example

* update: README for TMMLU+

* update: a small changes of TMMLU+ README file

* pre-commit run thought

* Add README for TMMLU+ dataset

* run precommit

* trigger precommit again

* trigger precommit again

* isort is fussy

* isort is fussy

* format, again

* oops

* oops

---------
Co-authored-by: default avatarlintang <lintang@eleuther.ai>
Co-authored-by: default avatarhaileyschoelkopf <hailey@eleuther.ai>
parent ff24e992
"dataset_name": "educational_psychology"
"description": "以下為教育心理的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_educational_psychology"
"task_alias": "educational psychology"
"dataset_name": "engineering_math"
"description": "以下為工程數學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_engineering_math"
"task_alias": "engineering math"
"dataset_name": "finance_banking"
"description": "以下為金融與法規的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_finance_banking"
"task_alias": "finance banking"
"dataset_name": "financial_analysis"
"description": "以下為財務分析的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_financial_analysis"
"task_alias": "financial analysis"
"dataset_name": "fire_science"
"description": "以下為火災學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_fire_science"
"task_alias": "fire science"
"dataset_name": "general_principles_of_law"
"description": "以下為法學大意的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_humanities"
"group_alias": "humanities"
"include": "_default_template_yaml"
"task": "tmmluplus_general_principles_of_law"
"task_alias": "general principles of law"
"dataset_name": "geography_of_taiwan"
"description": "以下為台灣地理的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_geography_of_taiwan"
"task_alias": "geography of taiwan"
"dataset_name": "human_behavior"
"description": "以下為人類行為與社會的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_human_behavior"
"task_alias": "human behavior"
"dataset_name": "insurance_studies"
"description": "以下為保險學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_insurance_studies"
"task_alias": "insurance studies"
"dataset_name": "introduction_to_law"
"description": "以下為法律概論的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_humanities"
"group_alias": "humanities"
"include": "_default_template_yaml"
"task": "tmmluplus_introduction_to_law"
"task_alias": "introduction to law"
"dataset_name": "jce_humanities"
"description": "以下為指考人文科目的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_humanities"
"group_alias": "humanities"
"include": "_default_template_yaml"
"task": "tmmluplus_jce_humanities"
"task_alias": "jce humanities"
"dataset_name": "junior_chemistry"
"description": "以下為國中理化的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_junior_chemistry"
"task_alias": "junior chemistry"
"dataset_name": "junior_chinese_exam"
"description": "以下為國中會考基測國文的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_junior_chinese_exam"
"task_alias": "junior chinese exam"
"dataset_name": "junior_math_exam"
"description": "以下為國中會考基測數學科的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_junior_math_exam"
"task_alias": "junior math exam"
"dataset_name": "junior_science_exam"
"description": "以下為國中會考基測自然科的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_junior_science_exam"
"task_alias": "junior science exam"
"dataset_name": "junior_social_studies"
"description": "以下為國中會考基測社會科的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_junior_social_studies"
"task_alias": "junior social studies"
"dataset_name": "linear_algebra"
"description": "以下為線代的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_linear_algebra"
"task_alias": "linear algebra"
"dataset_name": "logic_reasoning"
"description": "以下為邏輯思維的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_logic_reasoning"
"task_alias": "logic reasoning"
"dataset_name": "macroeconomics"
"description": "以下為總經的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_macroeconomics"
"task_alias": "macroeconomics"
"dataset_name": "management_accounting"
"description": "以下為管理會計的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_management_accounting"
"task_alias": "management accounting"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment