GitLab

MMLU task fix

Fix triviaqa task

Add - 4bit-related args

Add error handling for calling `.to(device)`

Fix non-callable attributes in CachingLM

Fix seqlen issues for bloom, remove extraneous OPT tokenizer check

Add --max_batch_size and --batch_size auto:N

Update evaluator.py cache_db argument str if model is not str

fixed the reference to the inner model

in case `isinstance(model, lm_eval.base.LM)` line 80 gives error, because `model` is not `str`. A simple inline if statement fixes this.

Revert "Add MultipleChoiceExactTask"

Add MultipleChoiceExactTask

* add anthropic support

* move requirement

* Fix tokenization issue in BaseLM.loglikelihood

* Add a regression script

* Use entire non-continuation length as context

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>