Skip to content

GitLab

Commit 23bd3449 authored by Yu Shi Jie's avatar Yu Shi Jie
Browse files

updated mmlu-pro to take on 3 splits: test, val, dev

parent 772d6f6f
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 3 deletions
lm_eval/tasks/mmlu_pro/default/_default_template_yaml
View file @ 23bd3449
dataset_path: sjyuxyz/MMLU-Pro-with-subset
test_split: test
fewshot_split: validation
fewshot_split: dev
fewshot_config:
sampler: first_n
output_type: multiple_choice
......
lm_eval/tasks/mmlu_pro/flan_cot_fewshot/_mmlu_pro_flan_cot_fewshot_template_yaml
View file @ 23bd3449
dataset_path: sjyuxyz/MMLU-Pro-with-subset
validation_split: validation
test_split: test
fewshot_split: validation
fewshot_split: dev
fewshot_config:
sampler: first_n
output_type: generate_until
......
lm_eval/tasks/mmlu_pro/generative/_default_template_yaml
View file @ 23bd3449
dataset_path: sjyuxyz/MMLU-Pro-with-subset
test_split: test
fewshot_split: validation
fewshot_split: dev
fewshot_config:
sampler: first_n
output_type: generate_until
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment