Unverified Commit fe44f98e authored by Lintang Sutawika's avatar Lintang Sutawika Committed by GitHub
Browse files

Update new_task_guide.md

parent cfc7a48b
...@@ -232,10 +232,6 @@ If you would like to run evaluation on all prompt templates, you can simply call ...@@ -232,10 +232,6 @@ If you would like to run evaluation on all prompt templates, you can simply call
use_prompt: "promptsource:*" use_prompt: "promptsource:*"
``` ```
### Weighting evaluation based on task size
By default, all tasks are aggregated by simple average (A group of 2 task with the same metric will simply be summed and divided by 2 for its group metric). You might find it necessary to aggregate multiple task scores by their weight. To do this, you can set within the task config `weight_by_size` to `True` to have its scores be weighted by the number of samples it has.
### Setting metrics ### Setting metrics
You're almost done! Now we need to choose how to score our task. You're almost done! Now we need to choose how to score our task.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment