GitLab

[Refactor] ALL_TASKS now maintained (not static)

fixes some minor issues on tasks. For yaml, the task name should be <dataset_path>_<dataset_name>:<task>

Better docs

[Refactor] Docs update

Merge branch 'big-refactor' of https://github.com/EleutherAI/lm-evaluation-harness into big-refactor

Remove the registration of "GPT2" as a model type

* add wip gsm8k yaml

* cleanup tasks dir

* push gsm8k yaml changes

* rename gpt2.py

* add updated gsm8k , triviaqa baseline

* add new cot yaml

* allow for multiple filter pipelines, new filter types

* updated gsm8k + sampling gen configs

* cleanup self-consistency yaml

* push outline for advanced docs

* push docs checklist

* switch to inheritance for many tasks

* acc_norm and acc_mutual_info fixed

* fix missing newline in error msg

* remove many .py tasks

* updated GSM8k

* added more doc

* Update advanced_task_guide.md

Added list of parameters

* Update advanced_task_guide.md

* Added details on listing metrics

* Update advanced_task_guide.md

* Added more explanation

* modify current default filter name

* add new tags to tasks

* remove a lingering print()

* add rest of param docs, cleanup deprecated fields

* push docs update

* move ALL_TASKS definition location

* confirm write_out.py works if no description dict passed

---------
Co-authored-by: lintangsutawika <lintang@sutawika.com>

Dataset metric log [WIP]

Merge branch 'dataset-metric-log' of github.com:EleutherAI/lm-evaluation-harness into dataset-metric-log

fixed typo

* add wip gsm8k yaml

* cleanup tasks dir

* push gsm8k yaml changes

* rename gpt2.py

* add updated gsm8k , triviaqa baseline

* add new cot yaml

* allow for multiple filter pipelines, new filter types

* updated gsm8k + sampling gen configs

* cleanup self-consistency yaml