Skip to content

GitLab

Commit 2aed1256 authored by Niklas Muennighoff's avatar Niklas Muennighoff
Browse files

Fix eval

parent 5ef6ea82
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
lm_eval/tasks/babi.py
View file @ 2aed1256
...@@ -74,7 +74,7 @@ class Babi(Task): ...@@ -74,7 +74,7 @@ class Babi(Task):
The results of the requests created in construct_requests. The results of the requests created in construct_requests.
""" """
gold = doc["answer"] gold = doc["answer"]
pred = gold == results pred = gold.strip() == results[0].strip()
return {"em": pred} return {"em": pred}
def aggregation(self): def aggregation(self):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment