Unverified Commit 8028a42f authored by Alexandre Marques's avatar Alexandre Marques Committed by GitHub
Browse files

Fixes to mmlu_pro_llama (#2816)

* Update generation_kwargs in default template to include additional end tokens

* Update filter_list in MMLU Pro configuration to use strict_match

* Update _default_template_yaml
parent c6b9aeeb
...@@ -19,9 +19,10 @@ metric_list: ...@@ -19,9 +19,10 @@ metric_list:
- "\\.$" - "\\.$"
generation_kwargs: generation_kwargs:
do_sample: false do_sample: false
until: temperature: 0
- "."
max_gen_toks: 1024 max_gen_toks: 1024
until:
- []
filter_list: filter_list:
- name: strict_match - name: strict_match
filter: filter:
......
...@@ -18,6 +18,6 @@ aggregate_metric_list: ...@@ -18,6 +18,6 @@ aggregate_metric_list:
- aggregation: mean - aggregation: mean
metric: exact_match metric: exact_match
weight_by_size: true weight_by_size: true
filter_list: custom-extract filter_list: [strict_match]
metadata: metadata:
version: 1.0 version: 1.0
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment