Skip to content

GitLab

Commit d78917a7 authored by Jon Tow's avatar Jon Tow
Browse files

Add note on accuracy reporting

parent 48ae5457
Show whitespace changes
Inline Side-by-side
Showing with 6 additions and 0 deletions
lm_eval/tasks/hendrycks_ethics.py
View file @ d78917a7
...@@ -8,6 +8,12 @@ from lm_eval.metrics import mean ...@@ -8,6 +8,12 @@ from lm_eval.metrics import mean
from lm_eval.utils import sh from lm_eval.utils import sh
from .common import yesno from .common import yesno
"""
NOTE: The reported "group" accuracies for the Deontology, Justice, and Virtue
tasks are refered to in this work as the `em` sub-metric. See Section 3. Metrics.
of the paper.
"""
class Ethics(Task): class Ethics(Task):
def download(self): def download(self):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment