Openllm benchmark (#1526)

8a875e9a · Baber Abbasi · GitHub · 292e5814 · 8a875e9a
Unverified Commit 8a875e9a authored Mar 05, 2024 by Baber Abbasi Committed by GitHub Mar 05, 2024
Hide whitespace changes
Inline Side-by-side

Showing with 18 additions and 0 deletions

lm_eval/tasks/benchmarks/openllm.yaml lm_eval/tasks/benchmarks/openllm.yaml +18 -0

No files found.
--- a/lm_eval/tasks/benchmarks/openllm.yaml
+++ b/lm_eval/tasks/benchmarks/openllm.yaml
+group: openllm
+group_alias: Open LLM Leaderboard
+task:
+  - task: arc_challenge
+    fewshot_split: validation
+    num_fewshot: 25
+  - task: hellaswag
+    fewshot_split: train
+    num_fewshot: 10
+  - task: truthfulqa
+    num_fewshot: 0
+  - task: mmlu
+    num_fewshot: 5
+  - task: winogrande
+    fewshot_split: train
+    num_fewshot: 5
+  - task: gsm8k
+    num_fewshot: 5