GitLab

* overhaul `ContextSampler`

* refactor masakhapos

* move multi_target to `exact_match`

* remove doc_to_choice from `boolq-seq2seq`

* remove doc_to_choice in generation process_results

* Remove unused `doc_to_choice` and fix superglue whitespaces

* require multiple_inputs and multiple_targets to be explicitly set in taskconfig

* fix copa; better logging in task init

* fix doc_to_target to return int rather than str (deprecated)

* fix processing regression; recursively parse lists fron template

* remove redundant jinja parsing logic

* remove promptsource

* for multiple_inputs use `doc_to_text: list[str]``

* Refactor `ContextSampler` `fewshot_context`

* fix multiple_input context

* fix `target_delimiter` with `gen_prefix`

* `doc_to_text` is list for multiple_inputs

* Refactor `count_bytes` and `count_words` methods to `@staticmethod`

* make has_*(train/test/validation) to properties

* remove `multi_target` `generate_until`

* `fix doc_to_target/multiple_targets handling add tests

* rename `multi_target` to `multiple_targets`

* evalaute list when multiple targets

* allow doc_to_target to return list

* Remove gen_prefix space and add warning (#3239)

* Remove gen_prefix space and add warning

* fix null gen_prefix bug again

* use git tests

---------
Co-authored-by: Boaz Ben-Dov <bendboaz@gmail.com>

* Add eqbench tasks in Spanish and Catalan

* Incremented catalan_bench and spanish_bench versions. Added 'multilingual' folder inside 'eq_bench' and moved the eqbench_ca and eqbench_es .yaml to that folder. Updated the tasks README with eqbench_es and eqbench_ca, expliciting inside each description both the Hugging Face link and the translation method.

* Fixed tasks table.

* remove test_task.sh and results folder

* Add utils.py to multilingual folder

* Add humaneval_infilling task

* pacify pre-commit

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* register aime

* lint

---------
Co-authored-by: Baber <baber@hey.com>

* create babilong tasks

* lint

* add clarification

* fix typo

* add babilong description

* Added benchmark

* Added more testing

* Added task definition for mmlu_redux and mmlu_redux_spanish

* Add MMLU Redux English and Spanish tasks with YAML fixes and READMEs

* Add remaining MMLU Redux YAMLs and updated tasks README

* Add MMLU Redux English and Spanish tasks with YAML fixes and READMEs

* Add MMLU Redux changes from pr-2705

* Resolve pre-commit hook and pytest overlapping group issues by adding mmlu_redux_spanish task entries and unique subgroup names

* Enhance retry logic to prevent 429 error when using Hugging Face API for tests, apply pre-commit fixes

* Revert python test changes and comments one task group to avoid Hugging Face rate limit and task failure

---------
Co-authored-by: CT-6282 <ricardo.godric@hotmail.com>

* fix: set 'do_sample=False' and use double quotes in 'doc_to_text'

* feat: update versions and README for longbench

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* add icelandic_winogrande

* fix spacing for final words in sentence

* Add EsBBQ and CaBBQ tasks

* Linter fixes

* add esbbq and cabbq to task list

---------
Co-authored-by: Júlia Falcão <juliafsfalcao@hotmail.com>

* run linter

* add acc_norm

* Fix codex-glue/code2text group issue

* Added README

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* add AIME tasks

* standardize the repeats

* fix task naming

* aime25 only has test set

* edit readme

* add utils

* standardize

* fix case sensitivity

* repeat once

* lint

* more linting

* lint huggingface.py

* update MMLU_ProX

* update MMLU_ProX

* cleanup code by pre-commit

* Anthropic Discrim Eval

* Mixed Effects Regression

* Actually wire it all upo

* Operator Name Doesn't Exist on Github

* Update lm_eval/tasks/discrim_eval/discrim_eval_implicit.yaml
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* Update discrim_eval_implicit.yaml

* Update discrim_eval_explicit.yaml

* pacify pre-commit

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>
Co-authored-by: Baber <baber@hey.com>

* feat: Add CLIcK task

* Fix formatting issues

* Add Click Task Description

* fix: lint

* fix

* update math_verify

* remove normalization

* use full solution in `parse`

* update version

Co-authored-by: Patrick Haller <phmaker@Patricks-MacBook-Pro.local>

* add lm_syneval

* edit readme

* update task readme

* formatting fixes

* run linting

* add descriptions and examples

* clean readme formatting

* add turblimp

* update general task readme

* add normalized accuracy

* add blimp_nl

* add template yaml file

* add zhoblimp files

* correct group name

* fix group

* add normalized accuracy

* Update afridiacritics_yaml

* Update afrisenti

* Update nollysenti

* Update ntrex

* Update salt

* improve include-path precedence handling

* test: add task for test

* add test for include path precedence handling

* Refactor `test_include_path.py`

---------
Co-authored-by: Baber <baber@hey.com>

The tasks are called without .yaml just the task name

* Update humaneval_64_instruct.yaml

Sync doc_to_text with humaneval_instruct.yaml

* Update humaneval_instruct.yaml

Remove redundant (flawed) spaces

* Update README.md

* Bump task version

* Update continuation group names to fit Readme

* added changelog to readme and switched datasets form hails to cais

* added missing new line at end of readme