merged with weight_by_size

docs/new_task_guide.md

@@ -232,6 +232,10 @@ If you would like to run evaluation on all prompt templates, you can simply call
 use_prompt: "promptsource:*"
 ```
 
+### Weighting evaluation based on task size
+
+By default, all tasks are aggregated by simple average (A group of 2 task with the same metric will simply be summed and divided by 2 for its group metric). You might find it necessary to aggregate multiple task scores by their weight. To do this, you can set within the task config `weight_by_size` to `True` to have its scores be weighted by the number of samples it has.
+
 ### Setting metrics
 
 You're almost done! Now we need to choose how to score our task.

lm_eval/api/task.py

@@ -95,7 +95,7 @@ class TaskConfig(dict):
     filter_list: Union[str, list] = None
     should_decontaminate: bool = False
     doc_to_decontamination_query: str = None
-
+    weight_by_size: bool = False
     metadata: Union[
         str, list
     ] = None  # by default, not used in the code. allows for users to pass arbitrary info to tasks

lm_eval/evaluator.py

@@ -485,7 +485,7 @@ def evaluate(
                         if "alias" in metrics:
                             metrics.pop("alias")
 
-                        if weight_by_size:
+                        if configs[task]["weight_by_size"]:
                             current_size = metrics.pop("samples")
                         else:
                             metrics.pop("samples")