GitLab

Sort descending as intended for `greedy_until`

[triviaqa] The ground truth must be a *substring* of the generated an…

add SCROLLS

MMLU task fix

Fix triviaqa task

Add - 4bit-related args

Add error handling for calling `.to(device)`

Fix non-callable attributes in CachingLM

Fix seqlen issues for bloom, remove extraneous OPT tokenizer check

Add --max_batch_size and --batch_size auto:N

Update evaluator.py cache_db argument str if model is not str

fixed the reference to the inner model

in case `isinstance(model, lm_eval.base.LM)` line 80 gives error, because `model` is not `str`. A simple inline if statement fixes this.

Revert "Add MultipleChoiceExactTask"

Add MultipleChoiceExactTask

* add anthropic support

* move requirement