Commit 44a602ab authored by haileyschoelkopf's avatar haileyschoelkopf
Browse files

add many explicit group configs

parent c9801daf
group:
tag:
- french_bench_perplexity
task: french_bench_wikitext_fr
dataset_path: asi/wikitext_fr
......
include: "_default_template_yaml"
group:
tag:
- french_bench
- french_bench_extra
description: "La prémisse et l'hypothèse sont elles en accord, neutres en elles, ou en contradiction ?"
......
......@@ -41,10 +41,14 @@ Homepage: https://gluebenchmark.com/
}
```
### Groups and Tasks
### Groups, Tags, and Tasks
#### Groups
None.
#### Tags
* `glue`: Run all Glue subtasks.
#### Tasks
......
group: glue
tag: glue
task: cola
dataset_path: glue
dataset_name: cola
......
group: glue
tag: glue
task: mnli
dataset_path: glue
dataset_name: mnli
......
group: glue
tag: glue
task: mrpc
dataset_path: glue
dataset_name: mrpc
......
group: glue
tag: glue
task: qnli
dataset_path: glue
dataset_name: qnli
......
group: glue
tag: glue
task: qqp
dataset_path: glue
dataset_name: qqp
......
group: glue
tag: glue
task: rte
dataset_path: glue
dataset_name: rte
......
group: glue
tag: glue
task: sst2
dataset_path: glue
dataset_name: sst2
......
group: glue
tag: glue
task: wnli
dataset_path: glue
dataset_name: wnli
......
......@@ -25,11 +25,15 @@ Homepage: `https://github.com/idavidrein/gpqa/tree/main`
This dataset is gated, so you will have to accept the terms of use at https://huggingface.co/datasets/Idavidrein/gpqa and login via `huggingface-cli login` using your HF Hub token before running this task.
### Groups and Tasks
### Groups, Tags, and Tasks
#### Groups
* `gpqa`
None
#### Tags
* `gpqa`: runs all GPQA variants.
#### Tasks
......
dataset_path: Idavidrein/gpqa
group: gpqa
tag: gpqa
output_type: generate_until
process_docs: !function utils.process_docs
training_split: train
......
dataset_path: Idavidrein/gpqa
group: gpqa
tag: gpqa
output_type: generate_until
process_docs: !function utils.process_docs
training_split: train
......
dataset_path: Idavidrein/gpqa
group: gpqa
tag: gpqa
output_type: generate_until
process_docs: !function utils.process_docs
training_split: train
......
dataset_path: Idavidrein/gpqa
group: gpqa
tag: gpqa
output_type: multiple_choice
process_docs: !function utils.process_docs
training_split: train
......
dataset_path: Idavidrein/gpqa
group: gpqa
tag: gpqa
output_type: multiple_choice
process_docs: !function utils.process_docs
training_split: train
......
group: haerae
dataset_path: HAERAE-HUB/HAE_RAE_BENCH
test_split: test
fewshot_split: test
......
group: haerae
task:
- haerae_gk
- haerae_hi
- haerae_lw
- haerae_rw
- haerae_sn
aggregate_metric_list:
- metric: acc
aggregation: mean
weight_by_size: true
- metric: acc_norm
aggregation: mean
weight_by_size: true
metadata:
version: 1.0
group:
- headqa
tag: headqa
task: headqa_en
dataset_path: EleutherAI/headqa
dataset_name: en
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment