"lm_eval/tasks/arab_culture_completion/utils_completion.py" did not exist on "5a481f4317ade9b12d2d9654f36a001f1a9192a6"
default.yml 402 Bytes
Newer Older
Stella Biderman's avatar
Stella Biderman committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
task: social_iqa
dataset_path: social_i_qa
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "Q: {{context}} {{question}}\nA:"
target_delimiter: " "
doc_to_choice: ["{{answerA}}", "{{answerB}}", "{{answerC}}"]
doc_to_target: "{{label}}"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
Stella Biderman's avatar
Stella Biderman committed
15
metadata:
16
  version: 0.0