Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
arc.yaml 351 Bytes
Newer Older
haileyschoelkopf's avatar
in-place replace main with lm-eval2, keeping old git history  
haileyschoelkopf committed
1
2
3
4
5
6
7 
dataset_path: ai2_arc
dataset_name: ARC-Challenge
training_split: train
validation_split: validation
test_split: test
doc_to_text: "Q: {{question}}\nA:"
doc_to_target: "{% set answer_choices = doc['choices']['text'] %}{{answer_choices[int(doc['answerKey']) - 1]}}"
lintangsutawika's avatar
updated yaml  
lintangsutawika committed
8
9
10
11 
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true