GitLab

* add Darija tasks

* fix multiple groups issue in darijammlu

* add MT to the description of the Darija tasks

* Update README.md

nit

* fix the recursion error caused by the darija_summarization task

* use a custom filter instead of the decorator for the strip function

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* Adding acpbench task

* adding ACPBench in Tasks readme.

* running precommit

* Multilingual MMLU

* Refactor process_docs function calls for clarity and consistency

* update mmlu_prox configs

* update tasks/README

* correct hyphon to underline in task/README

* update pre-commit codes

* Update generation_kwargs in default template to include additional end tokens

* Update filter_list in MMLU Pro configuration to use strict_match

* Update _default_template_yaml

* Update continuation template YAML for MMLU task with new generation and filtering options

* Refactor filter_list structure in continuation template YAML for improved readability

* Add 'take_first' function to filter_list in continuation template YAML

* Update filter_list in continuation template YAML to use 'strict_match' and modify filtering functions

* Add 'do_sample' option to generation_kwargs in MMLU template YAML

* add changelog to readme template

* add readme

* add to task list

suport for longcontext (and other synthetic tasks)
* add ruler
* add longbench
* pass `metadata` to TaskConfig

* add MastermindEval benchmark

* fill out checklist

* Add cocoteros_va dataset

* Fix format in cocoteros_va.yml

* Undo newline added

* Execute pre-commit to fix format errors

* Update catalan_bench.yaml version and add Changelog section into Readme.md

* Add INCLUDE tasks

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* Update openllm.yaml to use train fewshot split for arc

* Implementation of Winogender

* Minor fixes README.md

* Add winogender

* Clean winogender utils.py

* Change dataset to one containing All subsets

* Flesh out README for BBQ task

* Add missing tasks for BBQ

* Add simple cooccurrence bias task

* Fix wrong mask for ambiguated context+rename metrics

* Made generate_until evaluation (following PALM paper) default

Also moved separate config files per category to separate metrics using custom function.
Created config file for multiple_choice way of evaluating BBQ.

* Add missing version metadata

* Add missing versionmetadata for bbq multiple choice

* Fix metrics and address edge cases

* Made BBQ multiple choice the default version

* Added settings following winogrande

* Add num_fewshot to simple_cooccurrence_bias

* Fixes for bbq (multiple choice)

* Fix wrong dataset

* CrowS-Pairs: make it easier to use another dataset by removing dataset_name from the subsets.

* Use simplest prompt possible without description

* Merge

* BBQ: Fix np.NaN related bug

* BBQ: Fix wrong aggregation method for disamb accuracy

* BBQ: Make it possible to only evaluate on (dis)ambiguous subset (needed for few shot eval)

* BBQ: fix showing one target in case of few-shot evals

* BBQ: Fix few-shot example for bbq_generate

* BBQ: simplify subtasks

* BBQ: Minimize number of UNK variations to reduce inference time

* BBQ: Add extra UNK keywords for the generate task

* Add a generate_until version of simple_cooccurrence_bias

* Change system/description prompt to include few-shot examples

* Group agg rework

* Run pre-commit

* add tasks to readme table

* remove trailing space from simple_cooccurrence_bias_gen.yaml `doc_to_text`

* fix

* fix

* fix version

---------
Co-authored-by: Baber <baber@hey.com>

* Added audio-modality pipeline for qwen2-audio model

* Beauty imports

* fix apply_chat_template args

* update default audio placeholders list

* add demo task - common_voice subset

* add audiolm_qwen libs to pyproject.toml

* pre-commit beautify

---------
Co-authored-by: Alexandra Rak <rakalexandra@mail.ru>

* add instruct humaneval

* nit

* add to readme

* nit

* Capture gen_kwargs from CLI in squad_completion

* Update lm_eval/tasks/squad_completion/task.py
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* Update lm_eval/tasks/squad_completion/task.py
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* pre-commit

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>
Co-authored-by: Baber <baber@hey.com>

* New healthcare benchmark: careqa

* LAUNCH_MN5_ACC <python main.py --config config/mn5.yml --models Llama-3.2-1B-Instruct --tasks careqa_open --num_fewshot 0>

* Add fixes, READMES, and remove task_list.txt

* pre-commit passed, add formatting updates; add nanmean agg_metric

* Fix import error.

* Wrapped imports in try excepts

* Wrapped imports in try excepts; also metrics to catch bert_score import error

* Try except to catch ImportErrors as well

* use np.nan

* pre-commit

---------
Co-authored-by: PabloAgustin <pablo.martin@bsc.es>
Co-authored-by: Baber <baber@hey.com>

* fix for mc2 calculation

* increment versions and changelog

---------
Co-authored-by: Baber <baber@hey.com>

* Filter new leaderboard_math_hard dataset to "Level 5" only

* align to linters
Signed-off-by: Yotam Perlitz <y.perlitz@ibm.com>

---------
Signed-off-by: Yotam Perlitz <y.perlitz@ibm.com>

* Add a test for a custom unitxt task

* Update task.py to bring in line with breaking change in v1.17.2

* Fix lint

* Fix failing tests

* Resolved merge conflicts

* pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* remove ray.remote resources

* remove kobtest tag (registered as group)

* add humaneval+ and mbpp+

* add newline at end of file

* Fix the import source for eval_logger

* fix logging

---------
Co-authored-by: Baber <baber@hey.com>

Co-authored-by: Robiert Sepulveda Torres <rsepulveda911112@gmail.com>

* add Basque translation of ARC and PAWS to BasqueBench

* pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

Added IberoBench citation info (https://aclanthology.org/2025.coling-main.699/) in correpsonding READMEs (#2729)

* changed source of eval_logger

* allow eval_logger to be set from args

* removed verbosity arg from non-main methods

* fix logging

* pre-commit

* set verbosity in eval logger

* replace utils.eval_logger

* fix logging in main

* add logging to docs

* add logging message

* nit

* add logging to docs

* refactor setup_logging to utils

---------
Co-authored-by: Baber <baber@hey.com>

* add math_verify to minerva math

* add math_verify to benchmark

* fix error

* increment version

* set target delimiter to empty string

* nit

* add warning