GitLab

* feat(vllm_causallms): make collator ignore seed when splitting batch into chunks

* fix(collator): revert PR changes

* fix(vllm-causallm): update collator call with groupby None

* feat(sglang-causallms): make generation accept a list of sampling params

---------
Co-authored-by: Baber <baber@hey.com>

* add icelandic_winogrande

* fix spacing for final words in sentence

* Add EsBBQ and CaBBQ tasks

* Linter fixes

* add esbbq and cabbq to task list

---------
Co-authored-by: Júlia Falcão <juliafsfalcao@hotmail.com>

* run linter

* add acc_norm

* Fix codex-glue/code2text group issue

* Added README

* pacify pre-commit

---------
Co-authored-by: Baber <baber@hey.com>

* add AIME tasks

* standardize the repeats

* fix task naming

* aime25 only has test set

* edit readme

* add utils

* standardize

* fix case sensitivity

* repeat once

* lint

* more linting

* lint huggingface.py

* update MMLU_ProX

* update MMLU_ProX

* cleanup code by pre-commit

* Add support for OVModelForSeq2SeqLM

* Add test

* Anthropic Discrim Eval

* Mixed Effects Regression

* Actually wire it all upo

* Operator Name Doesn't Exist on Github

* Update lm_eval/tasks/discrim_eval/discrim_eval_implicit.yaml
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* Update discrim_eval_implicit.yaml

* Update discrim_eval_explicit.yaml

* pacify pre-commit

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>
Co-authored-by: Baber <baber@hey.com>

* feat: Add CLIcK task

* Fix formatting issues

* Add Click Task Description

* fix: lint

* fix

* update math_verify

* remove normalization

* use full solution in `parse`

* update version

Co-authored-by: Patrick Haller <phmaker@Patricks-MacBook-Pro.local>

* add lm_syneval

* edit readme

* update task readme

* formatting fixes

* run linting

* add descriptions and examples

* clean readme formatting

* add turblimp

* update general task readme

* add normalized accuracy

* add blimp_nl

* add template yaml file

* add zhoblimp files

* correct group name

* fix group

* add normalized accuracy

Adding support for OpenAI GPT-5 model; Models only support hardcoded tempeature=1 and stop=None (#3247)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Update afridiacritics_yaml

* Update afrisenti

* Update nollysenti

* Update ntrex

* Update salt

* improve include-path precedence handling

* test: add task for test

* add test for include path precedence handling

* Refactor `test_include_path.py`

---------
Co-authored-by: Baber <baber@hey.com>

The tasks are called without .yaml just the task name

* Update humaneval_64_instruct.yaml

Sync doc_to_text with humaneval_instruct.yaml

* Update humaneval_instruct.yaml

Remove redundant (flawed) spaces

* Update README.md

* Bump task version

* Update continuation group names to fit Readme

* added changelog to readme and switched datasets form hails to cais

* added missing new line at end of readme

* Update vLLM compatibility
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* add TokensPrompt to all generate calls

---------
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Baber <baber@hey.com>

* remove trust-remote-code

* add W605 rule

Device has been a deprecated arg for a few releases of vLLM and is now removed in 0.10.0 https://github.com/vllm-project/vllm/pull/21349

* Fix: pin datasets < 4.0

* fix

* update type hints in HF

* fix hellaswag path