Skip to content

GitLab

Unverified Commit 38a9c530 authored by Yongkeun Hwang's avatar Yongkeun Hwang Committed by GitHub
Browse files

fix: mmlu (generative) metric aggregation (#2761)

parent 6d2abda4
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 0 deletions
lm_eval/tasks/mmlu/generative/_mmlu.yaml
View file @ 38a9c530
......@@ -7,27 +7,32 @@ task:
aggregate_metric_list:
- metric: exact_match
weight_by_size: true
filter_list: get_response
- group: other
task:
- mmlu_other_generative
aggregate_metric_list:
- metric: exact_match
weight_by_size: true
filter_list: get_response
- group: social sciences
task:
- mmlu_social_sciences_generative
aggregate_metric_list:
- metric: exact_match
weight_by_size: true
filter_list: get_response
- group: humanities
task:
- mmlu_humanities_generative
aggregate_metric_list:
- metric: exact_match
weight_by_size: true
filter_list: get_response
aggregate_metric_list:
- aggregation: mean
metric: exact_match
weight_by_size: true
filter_list: get_response
metadata:
version: 3
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment