Group agg rework (#1741)

* add greoup_config arg

* add a group config that allows disabling table for group score and group aggregate in general

* fixed size configuration

* adjust config

* add group config

* adjust mmlu to use group_config

* fixed args input in aggregate_subtask_metrics

* fixed issues related to printing alias of group and updated yaml

* update all mmlu variants to include group_config

* edit format

* modify mmlu tasks

* adjust group to also be a configurable group

* add configurable group

* simplify get_task_list

* adjust group scoring with using ConfigurableGroup

* adjust args

* update mmlu

* update mmlu

* update to work with new group and task configuration

* readd group_agg

* readd files

* move prepare_print_tasks to evaluator_utils

* sort set to False by default, fix predict_only arg

* add version for groups

* reversed task list

* update additional condition when loading a group in a group yaml

* update truthfulqa

* add description regarding tags replacing group

* replace group to tag

* fixed conditional statement

* remove warning

* update loading of task group and newly added tags

* reformat with pre-commit

* fixed info log

* update

* fix bug

* fix bug

* use task id to differentiate tasks

* convert all groups to configurable groups

* use task_id

* reformat

* add task_id for python tasks as well

* add task_id for python tasks as well

* add task_id for python tasks as well

* revert truthfulqa

* revert mmlu tasks

* new mmlu config

* new group config parameter `tag_to_task`

* Update truthfulqa_mc2.yaml

* reformate

* add _process_group_config

* adjust task_id

* add get_subtask_list function to get proper subtask list

* group config to_dict update

* remove tag check

* update mmlu

* fix config passing issues

* add test yaml

* format fix

* add documentation

* corner case for single tag being called

* fix indentation

* formatting

* update all mmlu variants

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* remove group_alias

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* remove version for metadata

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* update mmlu/

* removed " " in make_table

* change how aggregate_metric is loaded

* change how aggregate_metric is loaded

* update aggregate_metric arg

* update format

* update format

* some docs fixes

* add groups for agieval, aexams, aclue

* add more explicit aggregation groups

* add more groupings / tags distinctions

* add more groupings

* more groupings

* add many explicit group configs

* add many explicit group configs

* add more explicit group configs

* add more explicit group configs

* add more error msgs, agg_metric -> agg_metric_list

* some docs updates

* update task_id to be updateable and uses group:task format

* make KMMLU a tag for now

* update docs

* don't duplicate task names

* fix merge conflicts?

* giving this a try

* clean up diff

* switch mmlu variants over to using

* don't use to-be-deprecated group: config field in overview notebook

* Python tasks which subclass ConfigurableTask now run

* update mmlu

* pre-commit format

* fixed sorting for multi-level printing

* move group api to separate file

* fix bbh aggregation filter usage

* track api/group.py

* adjust group and tags loading

* make explicit group configs for leaderboard and other newer tasks

* fix arabicmmlu

* update

* change arabicmmlu template name???

* update group alias

* fix printing bugs

* check table printing is correct ; update tests

* use mmlu_stem to have a group included in print tests

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>
Co-authored-by: haileyschoelkopf <hailey@eleuther.ai>

lm_eval/tasks/glue/README.md

@@ -41,10 +41,14 @@ Homepage: https://gluebenchmark.com/
 }
 ```
 
-### Groups and Tasks
+### Groups, Tags, and Tasks
 
 #### Groups
 
+None.
+
+#### Tags
+
 * `glue`: Run all Glue subtasks.
 
 #### Tasks

lm_eval/tasks/glue/cola/default.yaml

-group: glue
+tag: glue
 task: cola
 dataset_path: glue
 dataset_name: cola

lm_eval/tasks/glue/mnli/default.yaml

-group: glue
+tag: glue
 task: mnli
 dataset_path: glue
 dataset_name: mnli

lm_eval/tasks/glue/mrpc/default.yaml

-group: glue
+tag: glue
 task: mrpc
 dataset_path: glue
 dataset_name: mrpc

lm_eval/tasks/glue/qnli/default.yaml

-group: glue
+tag: glue
 task: qnli
 dataset_path: glue
 dataset_name: qnli

lm_eval/tasks/glue/qqp/default.yaml

-group: glue
+tag: glue
 task: qqp
 dataset_path: glue
 dataset_name: qqp

lm_eval/tasks/glue/rte/default.yaml

-group: glue
+tag: glue
 task: rte
 dataset_path: glue
 dataset_name: rte

lm_eval/tasks/glue/sst2/default.yaml

-group: glue
+tag: glue
 task: sst2
 dataset_path: glue
 dataset_name: sst2

lm_eval/tasks/glue/wnli/default.yaml

-group: glue
+tag: glue
 task: wnli
 dataset_path: glue
 dataset_name: wnli

lm_eval/tasks/gpqa/README.md

@@ -25,11 +25,15 @@ Homepage: `https://github.com/idavidrein/gpqa/tree/main`
 
 This dataset is gated, so you will have to accept the terms of use at https://huggingface.co/datasets/Idavidrein/gpqa and login via `huggingface-cli login` using your HF Hub token before running this task.
 
-### Groups and Tasks
+### Groups, Tags, and Tasks
 
 #### Groups
 
-* `gpqa`
+None
+
+#### Tags
+
+* `gpqa`: runs all GPQA variants.
 
 #### Tasks
 

lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml

 dataset_path: Idavidrein/gpqa
-group: gpqa
+tag: gpqa
 output_type: generate_until
 process_docs: !function utils.process_docs
 training_split: train

lm_eval/tasks/gpqa/cot_zeroshot/_gpqa_cot_zeroshot_yaml

 dataset_path: Idavidrein/gpqa
-group: gpqa
+tag: gpqa
 output_type: generate_until
 process_docs: !function utils.process_docs
 training_split: train

lm_eval/tasks/gpqa/generative/_gpqa_generative_n_shot_yaml

 dataset_path: Idavidrein/gpqa
-group: gpqa
+tag: gpqa
 output_type: generate_until
 process_docs: !function utils.process_docs
 training_split: train

lm_eval/tasks/gpqa/n_shot/_gpqa_n_shot_yaml

 dataset_path: Idavidrein/gpqa
-group: gpqa
+tag: gpqa
 output_type: multiple_choice
 process_docs: !function utils.process_docs
 training_split: train

lm_eval/tasks/gpqa/zeroshot/_gpqa_zeroshot_yaml

 dataset_path: Idavidrein/gpqa
-group: gpqa
+tag: gpqa
 output_type: multiple_choice
 process_docs: !function utils.process_docs
 training_split: train

lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml

 include: gsm8k-cot.yaml
-group:
+tag:
   - chain_of_thought
   - self_consistency
 task: gsm8k_cot_self_consistency

lm_eval/tasks/gsm8k/gsm8k-cot-zeroshot.yaml

-group:
+tag:
   - math_word_problems
 task: gsm8k_cot_zeroshot
 dataset_path: gsm8k

lm_eval/tasks/gsm8k/gsm8k-cot.yaml

@@ -61,7 +61,7 @@ generation_kwargs:
   - 'Q:'
   - </s>
   - <|im_end|>
-group:
+tag:
 - chain_of_thought
 metadata:
   version: 3.0

lm_eval/tasks/gsm8k/gsm8k.yaml

-group:
+tag:
   - math_word_problems
 task: gsm8k
 dataset_path: gsm8k

lm_eval/tasks/haerae/_default_haerae_yaml

-group: haerae
 dataset_path: HAERAE-HUB/HAE_RAE_BENCH
 test_split: test
 fewshot_split: test