"README_ORIGIN.md" did not exist on "1138f96c50cca43e1143ff4674d8e4c218bc37f0"
Unverified Commit c1dce09e authored by Stella Biderman's avatar Stella Biderman Committed by GitHub
Browse files

Create mmlu.yaml

Begin MMLU implementation
parent 6a000adb
group:
- multiple_choice
task: mmlu
dataset_path: cais/mmlu
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: test
template_aliases: "{% set answer_choices = [distractor1, distractor2, distractor3, correct_answer] %}{% set gold = 3 %}" # set the list of possible answer choices, and set what this doc's gold label idx is
doc_to_text: "{{support.lstrip()}}\nQuestion: {{question}}\nAnswer:"
doc_to_target: " {{correct_answer}}"
gold_alias: "{{gold}}" # this will be cast to an int.
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
- metric: acc_mutual_info
aggregation: mean
higher_is_better: true
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment