Title: `EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models`
Abstract: https://arxiv.org/abs/2312.06281
EQ-Bench is a benchmark for language models designed to assess emotional intelligence.
Why emotional intelligence? One reason is that it represents a subset of abilities that are important for the user experience, and which isn't explicitly tested by other benchmarks. Another reason is that it's not trivial to improve scores by fine tuning for the benchmark, which makes it harder to "game" the leaderboard.
EQ-Bench is a little different from traditional psychometric tests. It uses a specific question format, in which the subject has to read a dialogue then rate the intensity of possible emotional responses of one of the characters. Every question is interpretative and assesses the ability to predict the magnitude of the 4 presented emotions. The test is graded without the need for a judge (so there is no length bias). It's cheap to run (only 171 questions), and produces results that correlate strongly with human preference (Arena ELO) and multi-domain benchmarks like MMLU.
Homepage: https://eqbench.com/
NOTE: There are some key differences between the lm-evaluation-harness version and the implementation described in the EQ-Bench paper (These have been OK'd by the author):
- The lm-eval version uses the EQ-Bench v2 test set (171 questions) and score calculation. It does not incorporate the revision part of the prompt, as per v2.1 (https://github.com/EQ-bench/EQ-Bench)
- No retries in lm-eval version (EQ-Bench pipeline retries with successively higher temps if it encounters unparseable answers)
- In the original implementation, unparseable answers are excluded from the final score, and 83% of answers have to be parseable or a fail is returned. The lm-eval version instead assigns 0 to unparsable answers and has no fail criteria. So for lower performing models, there may be differences with the EQ-Bench leaderboard.
### Citation
```bibtex
@misc{paech2023eqbench,
title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models},
author={Samuel J. Paech},
year={2023},
eprint={2312.06281},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
### Groups and Tasks
#### Groups
* Not part of a group yet
#### Tasks
*`eq_bench`
### Checklist
* [x] Is the task an existing benchmark in the literature?
* [x] Have you referenced the original paper that introduced the task?
* [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
If other tasks on this dataset are already supported:
* [ ] Is the "Main" variant of this task clearly denoted?
* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?
FrenchBench is a benchmark for evaluating French language models, introduced in the paper
[CroissantLLM: A Truly Bilingual French-English Language Model](https://arxiv.org/abs/2402.00786).
It is a collection of tasks that evaluate the ability of a language model to understand and generate French text.
This benchmark is constructed both from openly available datasets, as well as newly released manually annotated data.
### Citation
```bibtex
@misc{faysse2024croissantllm,
title={CroissantLLM: A Truly Bilingual French-English Language Model},
author={Manuel Faysse and Patrick Fernandes and Nuno M. Guerreiro and António Loison and Duarte M. Alves and Caio Corro and Nicolas Boizard and João Alves and Ricardo Rei and Pedro H. Martins and Antoni Bigata Casademunt and François Yvon and André F. T. Martins and Gautier Viaud and Céline Hudelot and Pierre Colombo},
year={2024},
eprint={2402.00786},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
### Groups and Tasks
#### Groups
-`french_bench`: All tasks (non-perplexity based)
-`french_bench_gen`: All official generative tasks
-`french_bench_mc`: All official multiple choice tasks
-`french_bench_perplexity`: All perplexity-based tasks (0 shot is recommended)
-`french_bench_extra`: All extra tasks
#### Tasks
The following tasks evaluate tasks on the French Bench dataset using various scoring methods.
- french_bench_boolqa
- french_bench_fquadv2
- french_bench_fquadv2_bool
- french_bench_fquadv2_genq
- french_bench_fquadv2_hasAns
- french_bench_topic_based_nli
- french_bench_multifquad
- french_bench_grammar
- french_bench_vocab
- french_bench_reading_comp
- french_bench_xnli (modified XNLI)
- french_bench_orangesum_abstract
- french_bench_orangesum_title
- french_bench_trivia
- french_bench_hellaswag
- french_bench_arc_challenge
The french bench also includes other tasks from various benchmarks:
