Add Aggregation for Kobest Benchmark (#2446)

Co-authored-by: Baber <baber@hey.com>

Add Aggregation for Kobest Benchmark (#2446)
Co-authored-by: Baber <baber@hey.com>
94344a61 · Seungwoo Ryu · GitHub · 96e499ba · 94344a61
Unverified Commit 94344a61 authored Jan 29, 2025 by Seungwoo Ryu Committed by GitHub Jan 29, 2025
Hide whitespace changes
Inline Side-by-side

Showing with 19 additions and 0 deletions

lm_eval/tasks/kobest/_kobest.yaml lm_eval/tasks/kobest/_kobest.yaml +19 -0

No files found.
--- a/lm_eval/tasks/kobest/_kobest.yaml
+++ b/lm_eval/tasks/kobest/_kobest.yaml
+group: kobest
+task:
+  - kobest_boolq
+  - kobest_copa
+  - kobest_hellaswag
+  - kobest_sentineg
+  - kobest_wic
+aggregate_metric_list:
+  - metric: acc
+    aggregation: mean
+    weight_by_size: true
+  - metric: acc_norm
+    aggregation: mean
+    weight_by_size: true
+  - metric: f1
+    aggregation: mean
+    weight_by_size: true
+metadata:
+  version: 1.0