Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
tinyMMLU.yaml 382 Bytes
Newer Older
LucWeber's avatar
Add tinyBenchmarks  
LucWeber committed
1
2
3
4
5
6
7
8 
task: tinyMMLU
dataset_path: tinyBenchmarks/tinyMMLU
dataset_name: all
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
LucWeber's avatar
Run pre-commit  
LucWeber committed
9 
doc_to_text: "{{input_formatted}}"
LucWeber's avatar
Add tinyBenchmarks  
LucWeber committed
10
11
12
13
14
15
16
17
18 
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
num_fewshot: 0
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0