GitLab

* save progress

* fixed issue with table only showing 1 group

* store aliases directly in results_agg

* removed unused parts

* set `--gen_kwargs` to None + add help to CLI

* add logging metavar

* fix verbosity help messages

* Reorder severity levels.

* Add IFEval task

* Check and download nltk punkt if not already downloaded

* Update gen_max_toks to 2048 to support "900 words+" instructions

* Resolve pre-commit linting issues

* Reduce max_gen_toks to 1280 to conserve token usage

* Add warning message in `process_results` call for non chat-finetuned models

* fixed syntactic nits

* fix temperature and seed

* fix logprobs

* fixup merge

* enabling OpenAI completions via gooseai

* openai-completions and pin openai

* [fix] loading dataset from hub fails when the dataset name includes '.', as the program assumes it is on the local filesystem

* add FLD benchmark

* Update task.py

* [update] add group 'fld'

* [update] rename fld -> fld_default. add explanation to the readme

* Update README.md

---------
Co-authored-by: Lintang Sutawika <lintang@sutawika.com>

* fix: passing max_length to vllm engine args

* feat: add `max_model_len`

* chore: lint

* doc_to_decontamination_query can use function

* add option for doc_to_decontamination_query to follow doc_to_text

* added documentation for doc_to_decontamination_query

* adjust description

* format

* Additional process for doc_to_choice

* doc_to_choice can also parse a string

* modularize HFLM code

* pass through extra kwargs to AutoModel.from_pretrained call

* remove explicit model_kwargs

* rename gptq -> autogptq

* fix tokenizer pad token errors

* ensure model always respects device_map and autogptq's selected devices

* add a _get_config helper fn

* remove unlabled test sets

* add note to readme