GitLab

Add results of various models in json and md format

Create output path directory if necessary

* add xcopa dataset

* add xstory_cloze dataset and run pre-commit

* fix xcopa validation and test sets

* add xwinograd dataset

* add pawsx task

* add xnli task

* update task table with recently added tasks

* remove unused metrics from paws-x

* add mgsm task and fix gsm8k

* fix gsm8k until

* update task table

fix adaptive batch crash when there are no new requests

Set PAD token to EOS token

This makes comparing the results of different models easier because tasks are ordered in the same way.

Fix bugs introduced in #394 #406 and max length bug

Fix automatic batching for greedy_until
Fix stop sequences bug for greedy_until
Fix max length bug when model max length is not specified in tokenizer

single GPU automatic batching logic

Updated handling for device in lm_eval/models/gpt2.py

* Support bigbench-hard json tasks using multiple_choice_grade

* Add support for greedy decoding in bigbench tasks

* move bigbench_resources to datasets

* rectify changes to rf.greedy_until w upstream

* make path to resource import reflect new location

---------
Co-authored-by: haileyschoelkopf <hailey.schoelkopf@yale.edu>

Document task name wildcard support in README

Hotfix: patch issues with the `huggingface.py` model classes