Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
_mmlu.yaml 781 Bytes
Newer Older
Luis Cosio's avatar
feat: Add mmlu-redux and it's spanish transaltion as generative task definitions (#2705)  
Luis Cosio committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33 
group: mmlu_redux_generative
group_alias: mmlu_redux (generative)
task:
  - group: stem
    task:
      - mmlu_stem_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: true
  - group: other
    task:
      - mmlu_other_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: true
  - group: social sciences
    task:
      - mmlu_social_sciences_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: true
  - group: humanities
    task:
      - mmlu_humanities_generative
    aggregate_metric_list:
      - metric: exact_match
        weight_by_size: true
aggregate_metric_list:
  - aggregation: mean
    metric: exact_match
    weight_by_size: true
metadata:
  version: 3