Skip to content

GitLab

Unverified Commit 076372ee authored by Lintang Sutawika's avatar Lintang Sutawika Committed by GitHub
Browse files

Fix Issue regarding stderr (#1327) 

* add fix fordeciding if stderr is N/A or not

* process N/A
parent 181ccf43
Hide whitespace changes
Inline Side-by-side
Showing with 20 additions and 12 deletions
lm_eval/evaluator.py
View file @ 076372ee
...@@ -492,10 +492,13 @@ def evaluate( ...@@ -492,10 +492,13 @@ def evaluate(
]: ]:
stderr = "_stderr,".join(metric.split(",")) stderr = "_stderr,".join(metric.split(","))
stderr_score = results[task][stderr] stderr_score = results[task][stderr]
var_score = stderr_score**2 if stderr_score == "N/A":
metric_score = results[task][metric] var_score = "N/A"
else:
var_score = stderr_score**2
all_stderr.append(stderr)
all_stderr.append(stderr) metric_score = results[task][metric]
if metric in results[group]: if metric in results[group]:
results[group][metric] = ( results[group][metric] = (
...@@ -503,15 +506,20 @@ def evaluate( ...@@ -503,15 +506,20 @@ def evaluate(
+ metric_score * current_size + metric_score * current_size
) / (total_size + current_size) ) / (total_size + current_size)
# $$s_z^2 = \frac{(n-1) s_x^2 + (m-1) s_y^2}{n+m-1} + \frac{nm(\bar x - \bar y)^2}{(n+m)(n+m-1)}.$$ # $$s_z^2 = \frac{(n-1) s_x^2 + (m-1) s_y^2}{n+m-1} + \frac{nm(\bar x - \bar y)^2}{(n+m)(n+m-1)}.$$
results[group][stderr] = ( if var_score == "N/A":
(total_size - 1) * results[group][stderr] results[group][stderr] = "N/A"
+ (current_size - 1) * var_score else:
) / ( results[group][stderr] = (
total_size + current_size - 1 (total_size - 1) * results[group][stderr]
) + total_size * current_size / ( + (current_size - 1) * var_score
(total_size + current_size) ) / (
* (total_size + current_size - 1) total_size + current_size - 1
) * (results[group][metric] - metric_score) ** 2 ) + total_size * current_size / (
(total_size + current_size)
* (total_size + current_size - 1)
) * (
results[group][metric] - metric_score
) ** 2
else: else:
results[group][metric] = metric_score results[group][metric] = metric_score
results[group][stderr] = var_score results[group][stderr] = var_score
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment