GitLab

* create babilong tasks

* lint

* add clarification

* fix typo

* add babilong description

* Added benchmark

* Added more testing

* Added task definition for mmlu_redux and mmlu_redux_spanish

* Add MMLU Redux English and Spanish tasks with YAML fixes and READMEs

* Add remaining MMLU Redux YAMLs and updated tasks README

* Add MMLU Redux English and Spanish tasks with YAML fixes and READMEs

* Add MMLU Redux changes from pr-2705

* Resolve pre-commit hook and pytest overlapping group issues by adding mmlu_redux_spanish task entries and unique subgroup names

* Enhance retry logic to prevent 429 error when using Hugging Face API for tests, apply pre-commit fixes

* Revert python test changes and comments one task group to avoid Hugging Face rate limit and task failure

---------
Co-authored-by: CT-6282 <ricardo.godric@hotmail.com>

Signed-off-by: Liu, Kaixuan <kaixuan.liu@intel.com>

* fix: set 'do_sample=False' and use double quotes in 'doc_to_text'

* feat: update versions and README for longbench

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* feat(vllm_causallms): make collator ignore seed when splitting batch into chunks

* fix(collator): revert PR changes

* fix(vllm-causallm): update collator call with groupby None

* feat(sglang-causallms): make generation accept a list of sampling params

---------
Co-authored-by: Baber <baber@hey.com>

* add icelandic_winogrande

* fix spacing for final words in sentence

* Add EsBBQ and CaBBQ tasks

* Linter fixes

* add esbbq and cabbq to task list

---------
Co-authored-by: Júlia Falcão <juliafsfalcao@hotmail.com>

* run linter

* add acc_norm

* Fix codex-glue/code2text group issue

* Added README

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* add AIME tasks

* standardize the repeats

* fix task naming

* aime25 only has test set

* edit readme

* add utils

* standardize

* fix case sensitivity

* repeat once

* lint

* more linting

* lint huggingface.py

* update MMLU_ProX

* update MMLU_ProX

* cleanup code by pre-commit

* Add support for OVModelForSeq2SeqLM

* Add test

* Anthropic Discrim Eval

* Mixed Effects Regression

* Actually wire it all upo

* Operator Name Doesn't Exist on Github

* Update lm_eval/tasks/discrim_eval/discrim_eval_implicit.yaml
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* Update discrim_eval_implicit.yaml

* Update discrim_eval_explicit.yaml

* pacify pre-commit

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>
Co-authored-by: Baber <baber@hey.com>

* feat: Add CLIcK task

* Fix formatting issues

* Add Click Task Description

* fix: lint

* fix

* update math_verify

* remove normalization

* use full solution in `parse`

* update version

Co-authored-by: Patrick Haller <phmaker@Patricks-MacBook-Pro.local>

* add lm_syneval

* edit readme

* update task readme

* formatting fixes

* run linting

* add descriptions and examples

* clean readme formatting

* add turblimp

* update general task readme

* add normalized accuracy

* add blimp_nl

* add template yaml file

* add zhoblimp files

* correct group name

* fix group

* add normalized accuracy

Adding support for OpenAI GPT-5 model; Models only support hardcoded tempeature=1 and stop=None (#3247)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Update afridiacritics_yaml

* Update afrisenti

* Update nollysenti

* Update ntrex

* Update salt

* improve include-path precedence handling

* test: add task for test

* add test for include path precedence handling

* Refactor `test_include_path.py`

---------
Co-authored-by: Baber <baber@hey.com>

The tasks are called without .yaml just the task name

* Update humaneval_64_instruct.yaml

Sync doc_to_text with humaneval_instruct.yaml

* Update humaneval_instruct.yaml

Remove redundant (flawed) spaces

* Update README.md

* Bump task version

* Update continuation group names to fit Readme

* added changelog to readme and switched datasets form hails to cais

* added missing new line at end of readme

* Update vLLM compatibility
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* add TokensPrompt to all generate calls

---------
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Baber <baber@hey.com>