lm_eval/tasks/gpqa/zeroshot/gpqa_diamond_zeroshot.yaml · 9822b06e869b849fa45850b9ea2dcd3b0365e738 · gaoqiong / lm-evaluation-harness

Add a new task GPQA (the part without CoT) (#1434) · 5ab295c8

Uanu authored Feb 21, 2024



* add new task GPQA_n_shot

* add new task GPQA_zeroshot

* correct GPQA_zeroshot filename

* Add randomly shuffle choices

* Correct missing parentheses

* delete wrong tasks

* Add README

* Update lm_eval/tasks/gpqa/zeroshot/_gpqa_zeroshot_yaml

* Update lm_eval/tasks/gpqa/n_shot/utils.py

* Update lm_eval/tasks/gpqa/n_shot/utils.py

* Update lm_eval/tasks/gpqa/README.md

* placate linter

* linter

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

5ab295c8

gpqa_diamond_zeroshot.yaml 120 Bytes

Replace gpqa_diamond_zeroshot.yaml