Commit 44a602ab authored by haileyschoelkopf's avatar haileyschoelkopf
Browse files

add many explicit group configs

parent c9801daf
group: tag:
- french_bench_perplexity - french_bench_perplexity
task: french_bench_wikitext_fr task: french_bench_wikitext_fr
dataset_path: asi/wikitext_fr dataset_path: asi/wikitext_fr
......
include: "_default_template_yaml" include: "_default_template_yaml"
group: tag:
- french_bench - french_bench
- french_bench_extra - french_bench_extra
description: "La prémisse et l'hypothèse sont elles en accord, neutres en elles, ou en contradiction ?" description: "La prémisse et l'hypothèse sont elles en accord, neutres en elles, ou en contradiction ?"
......
...@@ -41,10 +41,14 @@ Homepage: https://gluebenchmark.com/ ...@@ -41,10 +41,14 @@ Homepage: https://gluebenchmark.com/
} }
``` ```
### Groups and Tasks ### Groups, Tags, and Tasks
#### Groups #### Groups
None.
#### Tags
* `glue`: Run all Glue subtasks. * `glue`: Run all Glue subtasks.
#### Tasks #### Tasks
......
group: glue tag: glue
task: cola task: cola
dataset_path: glue dataset_path: glue
dataset_name: cola dataset_name: cola
......
group: glue tag: glue
task: mnli task: mnli
dataset_path: glue dataset_path: glue
dataset_name: mnli dataset_name: mnli
......
group: glue tag: glue
task: mrpc task: mrpc
dataset_path: glue dataset_path: glue
dataset_name: mrpc dataset_name: mrpc
......
group: glue tag: glue
task: qnli task: qnli
dataset_path: glue dataset_path: glue
dataset_name: qnli dataset_name: qnli
......
group: glue tag: glue
task: qqp task: qqp
dataset_path: glue dataset_path: glue
dataset_name: qqp dataset_name: qqp
......
group: glue tag: glue
task: rte task: rte
dataset_path: glue dataset_path: glue
dataset_name: rte dataset_name: rte
......
group: glue tag: glue
task: sst2 task: sst2
dataset_path: glue dataset_path: glue
dataset_name: sst2 dataset_name: sst2
......
group: glue tag: glue
task: wnli task: wnli
dataset_path: glue dataset_path: glue
dataset_name: wnli dataset_name: wnli
......
...@@ -25,11 +25,15 @@ Homepage: `https://github.com/idavidrein/gpqa/tree/main` ...@@ -25,11 +25,15 @@ Homepage: `https://github.com/idavidrein/gpqa/tree/main`
This dataset is gated, so you will have to accept the terms of use at https://huggingface.co/datasets/Idavidrein/gpqa and login via `huggingface-cli login` using your HF Hub token before running this task. This dataset is gated, so you will have to accept the terms of use at https://huggingface.co/datasets/Idavidrein/gpqa and login via `huggingface-cli login` using your HF Hub token before running this task.
### Groups and Tasks ### Groups, Tags, and Tasks
#### Groups #### Groups
* `gpqa` None
#### Tags
* `gpqa`: runs all GPQA variants.
#### Tasks #### Tasks
......
dataset_path: Idavidrein/gpqa dataset_path: Idavidrein/gpqa
group: gpqa tag: gpqa
output_type: generate_until output_type: generate_until
process_docs: !function utils.process_docs process_docs: !function utils.process_docs
training_split: train training_split: train
......
dataset_path: Idavidrein/gpqa dataset_path: Idavidrein/gpqa
group: gpqa tag: gpqa
output_type: generate_until output_type: generate_until
process_docs: !function utils.process_docs process_docs: !function utils.process_docs
training_split: train training_split: train
......
dataset_path: Idavidrein/gpqa dataset_path: Idavidrein/gpqa
group: gpqa tag: gpqa
output_type: generate_until output_type: generate_until
process_docs: !function utils.process_docs process_docs: !function utils.process_docs
training_split: train training_split: train
......
dataset_path: Idavidrein/gpqa dataset_path: Idavidrein/gpqa
group: gpqa tag: gpqa
output_type: multiple_choice output_type: multiple_choice
process_docs: !function utils.process_docs process_docs: !function utils.process_docs
training_split: train training_split: train
......
dataset_path: Idavidrein/gpqa dataset_path: Idavidrein/gpqa
group: gpqa tag: gpqa
output_type: multiple_choice output_type: multiple_choice
process_docs: !function utils.process_docs process_docs: !function utils.process_docs
training_split: train training_split: train
......
group: haerae
dataset_path: HAERAE-HUB/HAE_RAE_BENCH dataset_path: HAERAE-HUB/HAE_RAE_BENCH
test_split: test test_split: test
fewshot_split: test fewshot_split: test
......
group: haerae
task:
- haerae_gk
- haerae_hi
- haerae_lw
- haerae_rw
- haerae_sn
aggregate_metric_list:
- metric: acc
aggregation: mean
weight_by_size: true
- metric: acc_norm
aggregation: mean
weight_by_size: true
metadata:
version: 1.0
group: tag: headqa
- headqa
task: headqa_en task: headqa_en
dataset_path: EleutherAI/headqa dataset_path: EleutherAI/headqa
dataset_name: en dataset_name: en
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment