Unverified Commit 9ae96cdf authored by ZoneTwelve's avatar ZoneTwelve Committed by GitHub
Browse files

TMMLU+ implementation (#1394)



* implementation of TMMLU+

* implemented: TMMLU+

****TMMLU+ : large-scale Traditional chinese Massive Multitask language Understanding****

- 4 categories
    - STEM
    - Social Science
    - Humanities
    - Other

The TMMLU+ dataset, encompassing over 67 subjects and 20160 tasks, is six times larger and more balanced than its predecessor, TMMLU, and includes benchmark results from both closed-source and 20 open-weight Chinese large language models with 1.8B to 72B parameters. However, Traditional Chinese variants continue to underperform compared to major Simplified Chinese models.

```markdown
Total number of tasks in the 'test' sets: 20160
Total number of tasks in the 'validation' sets: 2247
Total number of tasks in the 'train' sets: 335
```

* Remove print from __init__.py

There was my mistake in forgetting to remove the debug print from the code.

* update: move TMMLU+ config generation program into default

* fix: we should use training set as few shots example

* update: README for TMMLU+

* update: a small changes of TMMLU+ README file

* pre-commit run thought

* Add README for TMMLU+ dataset

* run precommit

* trigger precommit again

* trigger precommit again

* isort is fussy

* isort is fussy

* format, again

* oops

* oops

---------
Co-authored-by: default avatarlintang <lintang@eleuther.ai>
Co-authored-by: default avatarhaileyschoelkopf <hailey@eleuther.ai>
parent ff24e992
"dataset_name": "marketing_management"
"description": "以下為行銷管理的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_marketing_management"
"task_alias": "marketing management"
"dataset_name": "mechanical"
"description": "以下為機械與機電概論的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_mechanical"
"task_alias": "mechanical"
"dataset_name": "music"
"description": "以下為音樂科的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_music"
"task_alias": "music"
"dataset_name": "national_protection"
"description": "以下為軍事的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_national_protection"
"task_alias": "national protection"
"dataset_name": "nautical_science"
"description": "以下為航海的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_nautical_science"
"task_alias": "nautical science"
"dataset_name": "occupational_therapy_for_psychological_disorders"
"description": "以下為心理障礙職能治療學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_occupational_therapy_for_psychological_disorders"
"task_alias": "occupational therapy for psychological disorders"
"dataset_name": "official_document_management"
"description": "以下為機關文書的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_official_document_management"
"task_alias": "official document management"
"dataset_name": "optometry"
"description": "以下為視光學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_optometry"
"task_alias": "optometry"
"dataset_name": "organic_chemistry"
"description": "以下為有機化學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_organic_chemistry"
"task_alias": "organic chemistry"
"dataset_name": "pharmacology"
"description": "以下為藥理學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_pharmacology"
"task_alias": "pharmacology"
"dataset_name": "pharmacy"
"description": "以下為藥劑學的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_pharmacy"
"task_alias": "pharmacy"
"dataset_name": "physical_education"
"description": "以下為體育的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_physical_education"
"task_alias": "physical education"
"dataset_name": "physics"
"description": "以下為物理的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_physics"
"task_alias": "physics"
"dataset_name": "politic_science"
"description": "以下為政治的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_politic_science"
"task_alias": "politic science"
"dataset_name": "real_estate"
"description": "以下為房地產的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_real_estate"
"task_alias": "real estate"
"dataset_name": "secondary_physics"
"description": "以下為高中物理的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_secondary_physics"
"task_alias": "secondary physics"
"dataset_name": "statistics_and_machine_learning"
"description": "以下為統計與機器學習的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_STEM"
"group_alias": "STEM"
"include": "_default_template_yaml"
"task": "tmmluplus_statistics_and_machine_learning"
"task_alias": "statistics and machine learning"
"dataset_name": "taiwanese_hokkien"
"description": "以下為閩南語的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_social_sciences"
"group_alias": "social sciences"
"include": "_default_template_yaml"
"task": "tmmluplus_taiwanese_hokkien"
"task_alias": "taiwanese hokkien"
"dataset_name": "taxation"
"description": "以下為稅務的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_humanities"
"group_alias": "humanities"
"include": "_default_template_yaml"
"task": "tmmluplus_taxation"
"task_alias": "taxation"
"dataset_name": "technical"
"description": "以下為技術工相關的單選題,請提供正確答案的選項。\n\n"
"group": "tmmluplus_other"
"group_alias": "other"
"include": "_default_template_yaml"
"task": "tmmluplus_technical"
"task_alias": "technical"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment