lm_eval/tasks/jsonschema_bench/jsonschema_bench_hard.yaml · 6cc41d3452706c4a79d5221df2546e8089487125 · gaoqiong / lm-evaluation-harness

Add JSONSchemaBench: A Benchmark for Evaluating Structured Output from LLMs (#2865) · 6cc41d34

Saibo-creator authored Apr 02, 2025



* Add JSON schema benchmark

* Update lm_eval/tasks/jsonschema_bench/metrics.py

Thanks for catching this
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

* run pre-commit

* add description to task catalogue readme

---------
Co-authored-by: Baber Abbasi <92168766+baberabb@users.noreply.github.com>

6cc41d34

jsonschema_bench_hard.yaml 90 Bytes

Replace jsonschema_bench_hard.yaml