Commit 5251525d authored by haileyschoelkopf's avatar haileyschoelkopf
Browse files

some docs updates

parent 93c17c57
...@@ -331,7 +331,10 @@ task: ...@@ -331,7 +331,10 @@ task:
- cb - cb
- anli_r1 - anli_r1
- rte - rte
aggregate_metric: true aggregate_metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- task: mmlu - task: mmlu
num_fewshot: 2 num_fewshot: 2
``` ```
...@@ -340,6 +343,8 @@ It's also important to note how you can basically insert a group config as a tas ...@@ -340,6 +343,8 @@ It's also important to note how you can basically insert a group config as a tas
### Duplicate Tasks in Group Configs ### Duplicate Tasks in Group Configs
There might be cases where you might want to evaluate prompts and how models perform over prompt variations. You can list an existing task (In the example below, `anli_r1`) which varying `doc_to_text` implementation. To differentiate from each variation, we can utilize `task_alias`. LM-Eval will recognize that there are multiple variations of the same tasks and differentiate them. There might be cases where you might want to evaluate prompts and how models perform over prompt variations. You can list an existing task (In the example below, `anli_r1`) which varying `doc_to_text` implementation. To differentiate from each variation, we can utilize `task_alias`. LM-Eval will recognize that there are multiple variations of the same tasks and differentiate them.
```yaml ```yaml
group: flan_held_in group: flan_held_in
group_alias: Flan (Held-In) group_alias: Flan (Held-In)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment