Group agg rework (#1741)

* add greoup_config arg

* add a group config that allows disabling table for group score and group aggregate in general

* fixed size configuration

* adjust config

* add group config

* adjust mmlu to use group_config

* fixed args input in aggregate_subtask_metrics

* fixed issues related to printing alias of group and updated yaml

* update all mmlu variants to include group_config

* edit format

* modify mmlu tasks

* adjust group to also be a configurable group

* add configurable group

* simplify get_task_list

* adjust group scoring with using ConfigurableGroup

* adjust args

* update mmlu

* update mmlu

* update to work with new group and task configuration

* readd group_agg

* readd files

* move prepare_print_tasks to evaluator_utils

* sort set to False by default, fix predict_only arg

* add version for groups

* reversed task list

* update additional condition when loading a group in a group yaml

* update truthfulqa

* add description regarding tags replacing group

* replace group to tag

* fixed conditional statement

* remove warning

* update loading of task group and newly added tags

* reformat with pre-commit

* fixed info log

* update

* fix bug

* fix bug

* use task id to differentiate tasks

* convert all groups to configurable groups

* use task_id

* reformat

* add task_id for python tasks as well

* add task_id for python tasks as well

* add task_id for python tasks as well

* revert truthfulqa

* revert mmlu tasks

* new mmlu config

* new group config parameter `tag_to_task`

* Update truthfulqa_mc2.yaml

* reformate

* add _process_group_config

* adjust task_id

* add get_subtask_list function to get proper subtask list

* group config to_dict update

* remove tag check

* update mmlu

* fix config passing issues

* add test yaml

* format fix

* add documentation

* corner case for single tag being called

* fix indentation

* formatting

* update all mmlu variants

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* remove group_alias

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* remove version for metadata

* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* update mmlu/

* removed " " in make_table

* change how aggregate_metric is loaded

* change how aggregate_metric is loaded

* update aggregate_metric arg

* update format

* update format

* some docs fixes

* add groups for agieval, aexams, aclue

* add more explicit aggregation groups

* add more groupings / tags distinctions

* add more groupings

* more groupings

* add many explicit group configs

* add many explicit group configs

* add more explicit group configs

* add more explicit group configs

* add more error msgs, agg_metric -> agg_metric_list

* some docs updates

* update task_id to be updateable and uses group:task format

* make KMMLU a tag for now

* update docs

* don't duplicate task names

* fix merge conflicts?

* giving this a try

* clean up diff

* switch mmlu variants over to using

* don't use to-be-deprecated group: config field in overview notebook

* Python tasks which subclass ConfigurableTask now run

* update mmlu

* pre-commit format

* fixed sorting for multi-level printing

* move group api to separate file

* fix bbh aggregation filter usage

* track api/group.py

* adjust group and tags loading

* make explicit group configs for leaderboard and other newer tasks

* fix arabicmmlu

* update

* change arabicmmlu template name???

* update group alias

* fix printing bugs

* check table printing is correct ; update tests

* use mmlu_stem to have a group included in print tests

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>
Co-authored-by: haileyschoelkopf <hailey@eleuther.ai>

lm_eval/tasks/haerae/_haerae.yaml

+group: haerae
+task:
+  - haerae_gk
+  - haerae_hi
+  - haerae_lw
+  - haerae_rw
+  - haerae_sn
+aggregate_metric_list:
+  - metric: acc
+    aggregation: mean
+    weight_by_size: true
+  - metric: acc_norm
+    aggregation: mean
+    weight_by_size: true
+metadata:
+  version: 1.0

lm_eval/tasks/headqa/headqa_en.yaml

-group:
-  - headqa
+tag: headqa
 task: headqa_en
 dataset_path: EleutherAI/headqa
 dataset_name: en

lm_eval/tasks/hellaswag/hellaswag.yaml

-group:
+tag:
   - multiple_choice
 task: hellaswag
 dataset_path: hellaswag

lm_eval/tasks/hendrycks_ethics/commonsense.yaml

-group:
+tag:
   - hendrycks_ethics
 task: ethics_cm
 dataset_path: EleutherAI/hendrycks_ethics

lm_eval/tasks/hendrycks_ethics/justice.yaml

 include: deontology.yaml
-group:
+tag:
   - hendrycks_ethics
 task: ethics_justice
 dataset_name: justice

lm_eval/tasks/hendrycks_ethics/utilitarianism.yaml

 include: commonsense.yaml
-group:
+tag:
   - hendrycks_ethics
 task: ethics_utilitarianism
 dataset_name: utilitarianism

lm_eval/tasks/hendrycks_ethics/virtue.yaml

 include: commonsense.yaml
-group:
+tag:
   - hendrycks_ethics
 task: ethics_virtue
 dataset_name: virtue

lm_eval/tasks/hendrycks_math/hendrycks_math.yaml

@@ -7,3 +7,9 @@ task:
   - hendrycks_math_num_theory
   - hendrycks_math_prealgebra
   - hendrycks_math_precalc
+aggregate_metric_list:
+  - metric: exact_match
+    aggregation: mean
+    weight_by_size: true
+metadata:
+  version: 1.0

lm_eval/tasks/hendrycks_math/hendrycks_math_algebra.yaml

-group:
+tag:
   - math_word_problems
 task: hendrycks_math_algebra
 dataset_path: EleutherAI/hendrycks_math

lm_eval/tasks/inverse_scaling/_inverse_scaling_mc_yaml

-group:
+tag:
   - inverse_scaling_mc
 output_type: multiple_choice
 test_split: train

lm_eval/tasks/inverse_scaling/_some_results

+# |                   Tasks                   |Version|Filter|n-shot| Metric |Value |   |Stderr|
+# |-------------------------------------------|-------|------|-----:|--------|-----:|---|-----:|
+# | - inverse_scaling_hindsight_neglect_10shot|      0|none  |     0|acc     |0.4476|±  |0.0281|
+# |                                           |       |none  |     0|acc_norm|0.4476|±  |0.0281|
+# |inverse_scaling_mc                         |N/A    |none  |     0|acc_norm|0.6273|±  |0.0096|
+# |                                           |       |none  |     0|acc     |0.6210|±  |0.0095|
+# | - inverse_scaling_neqa                    |      0|none  |     0|acc     |0.5300|±  |0.0289|
+# |                                           |       |none  |     0|acc_norm|0.5300|±  |0.0289|
+# | - inverse_scaling_quote_repetition        |      0|none  |     0|acc     |0.9367|±  |0.0141|
+# |                                           |       |none  |     0|acc_norm|0.9367|±  |0.0141|
+# | - inverse_scaling_redefine_math           |      0|none  |     0|acc     |0.7178|±  |0.0150|
+# |                                           |       |none  |     0|acc_norm|0.7178|±  |0.0150|
+# | - inverse_scaling_winobias_antistereotype |      0|none  |     0|acc     |0.3786|±  |0.0239|
+# |                                           |       |none  |     0|acc_norm|0.4126|±  |0.0243|
+
+# |      Groups      |Version|Filter|n-shot| Metric |Value |   |Stderr|
+# |------------------|-------|------|-----:|--------|-----:|---|-----:|
+# |inverse_scaling_mc|N/A    |none  |     0|acc_norm|0.6273|±  |0.0096|
+# |                  |       |none  |     0|acc     |0.6210|±  |0.0095|
+# hf (pretrained=facebook/opt-2.7b,add_bos_token=True,dtype=float32), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
+# |                   Tasks                   |Version|Filter|n-shot| Metric |Value |   |Stderr|
+# |-------------------------------------------|-------|------|-----:|--------|-----:|---|-----:|
+# | - inverse_scaling_hindsight_neglect_10shot|      0|none  |     0|acc     |0.4476|±  |0.0281|
+# |                                           |       |none  |     0|acc_norm|0.4476|±  |0.0281|
+# |inverse_scaling_mc                         |N/A    |none  |     0|acc_norm|0.6291|±  |0.0095|
+# |                                           |       |none  |     0|acc     |0.6219|±  |0.0095|
+# | - inverse_scaling_neqa                    |      0|none  |     0|acc     |0.5267|±  |0.0289|
+# |                                           |       |none  |     0|acc_norm|0.5267|±  |0.0289|
+# | - inverse_scaling_quote_repetition        |      0|none  |     0|acc     |0.9433|±  |0.0134|
+# |                                           |       |none  |     0|acc_norm|0.9433|±  |0.0134|
+# | - inverse_scaling_redefine_math           |      0|none  |     0|acc     |0.7200|±  |0.0150|
+# |                                           |       |none  |     0|acc_norm|0.7200|±  |0.0150|
+# | - inverse_scaling_winobias_antistereotype |      0|none  |     0|acc     |0.3762|±  |0.0239|
+# |                                           |       |none  |     0|acc_norm|0.4150|±  |0.0243|
+
+# |      Groups      |Version|Filter|n-shot| Metric |Value |   |Stderr|
+# |------------------|-------|------|-----:|--------|-----:|---|-----:|
+# |inverse_scaling_mc|N/A    |none  |     0|acc_norm|0.6291|±  |0.0095|
+# |                  |       |none  |     0|acc     |0.6219|±  |0.0095|

lm_eval/tasks/kmmlu/cot_hard/_cot_kmmlu_yaml

-group:
+tag:
     - kmmlu
     - kmmlu_hard_cot
 dataset_path: HAERAE-HUB/KMMLU-HARD

lm_eval/tasks/kmmlu/direct/_direct_kmmlu_yaml

-group:
+tag:
     - kmmlu
     - kmmlu_direct
 dataset_path: HAERAE-HUB/KMMLU

lm_eval/tasks/kmmlu/direct_hard/_direct_hard_kmmlu_yaml

-group:
+tag:
     - kmmlu
     - kmmlu_hard_direct
 dataset_path: HAERAE-HUB/KMMLU-HARD

lm_eval/tasks/kmmlu/hard/_hard_kmmlu_yaml

-group:
+tag:
     - kmmlu
     - kmmlu_hard
 dataset_path: HAERAE-HUB/KMMLU-HARD

lm_eval/tasks/kobest/kobest_boolq.yaml

-group:
+tag:
   - kobest
 task: kobest_boolq
 dataset_path: skt/kobest_v1

lm_eval/tasks/kobest/kobest_copa.yaml

-group:
+tag:
   - kobest
 task: kobest_copa
 dataset_path: skt/kobest_v1

lm_eval/tasks/kobest/kobest_hellaswag.yaml

-group:
+tag:
   - kobest
 task: kobest_hellaswag
 dataset_path: skt/kobest_v1

lm_eval/tasks/kobest/kobest_sentineg.yaml

-group:
+tag:
   - kobest
 task: kobest_sentineg
 dataset_path: skt/kobest_v1

lm_eval/tasks/kobest/kobest_wic.yaml

-group:
+tag:
   - kobest
 task: kobest_wic
 dataset_path: skt/kobest_v1