GitLab

Add --max_batch_size and --batch_size auto:N

Update evaluator.py cache_db argument str if model is not str

fixed the reference to the inner model

in case `isinstance(model, lm_eval.base.LM)` line 80 gives error, because `model` is not `str`. A simple inline if statement fixes this.

Revert "Add MultipleChoiceExactTask"

Add MultipleChoiceExactTask

* add anthropic support

* move requirement

* Fix tokenization issue in BaseLM.loglikelihood

* Add a regression script

* Use entire non-continuation length as context

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

Add support for loading GPTQ models via AutoGPTQ

* allow for hf-causal to take dtype arg

* document this change

Add perplexity task on arbitrary JSON data

Add option to dump prompts and completions to a JSON file

Evaluation Against Portion of Benchmark Data

Add results of various models in json and md format

Create output path directory if necessary

* add xcopa dataset

* add xstory_cloze dataset and run pre-commit

* fix xcopa validation and test sets

* add xwinograd dataset

* add pawsx task

* add xnli task

* update task table with recently added tasks

* remove unused metrics from paws-x

* add mgsm task and fix gsm8k

* fix gsm8k until

* update task table