Truthfulqa multi harness (#3062)

* truthfulqa-multi task

* truthfulqa-multi with chat few-shot

* few shot chat implementation

* changed until so it outputs lists

* changed dataset location

* added MT task

* Create README.md

* do not include MT

* changes for PR

* tag change

* removed yaml extension

* adding task to the table

* fix task configs

* add import exception

---------
Co-authored-by: Baber <baber@hey.com>

lm_eval/tasks/README.md

@@ -150,6 +150,7 @@
 | [translation](translation/README.md)                                     | Tasks focused on evaluating the language translation capabilities of models.                                                                                                                                                                                                                                                           | Arabic, English, Spanish, Basque, Hindi, Indonesian, Burmese, Russian, Swahili, Telugu, Chinese                       |
 | [triviaqa](triviaqa/README.md)                                           | A large-scale dataset for trivia question answering to test general knowledge.                                                                                                                                                                                                                                                         | English                                                                                                               |
 | [truthfulqa](truthfulqa/README.md)                                       | A QA task aimed at evaluating the truthfulness and factual accuracy of model responses.                                                                                                                                                                                                                                                | English                                                                                                               |
+| [truthfulqa-multi](truthfulqa-multi/README.md) | Is a multilingual version of TruthfulQA, a QA task aimed at evaluating the truthfulness and factual accuracy of model responses. | English, Spanish, Catalan, Basque, Galician |
 | [turkishmmlu](turkishmmlu/README.md)                                     | A multiple-choice QA test modeled after MMLU, written in Turkish based on Turkish high-school level exams.                                                                                                                                                                                                                             | Turkish                                                                                                               |
 | [unitxt](unitxt/README.md)                                               | A number of tasks implemented using the unitxt library for flexible, shareable, and reusable data preparation and evaluation for generative AI.                                                                                                                                                                                        | English                                                                                                               |
 | [unscramble](unscramble/README.md)                                       | Tasks involving the rearrangement of scrambled sentences to test syntactic understanding.                                                                                                                                                                                                                                              | English                                                                                                               |

lm_eval/tasks/truthfulqa-multi/README.md

+# TruthfulQA-Multi
+
+## Paper
+
+Title: `Truth Knows No Language: Evaluating Truthfulness Beyond English`
+
+Abstract: `[https://arxiv.org/abs/2502.09387v1](https://arxiv.org/abs/2502.09387v1)`
+
+We introduce a professionally translated extension of the TruthfulQA benchmark designed to evaluate truthfulness in Basque, Catalan, Galician, and Spanish. Truthfulness evaluations of large language models (LLMs) have primarily been conducted in English. However, the ability of LLMs to maintain truthfulness across languages remains under-explored. Our study evaluates 12 state-of-the-art open LLMs, comparing base and instruction-tuned models using human evaluation, multiple-choice metrics, and LLM-as-a-Judge scoring. Our findings reveal that, while LLMs perform best in English and worst in Basque (the lowest-resourced language), overall truthfulness discrepancies across languages are smaller than anticipated. Furthermore, we show that LLM-as-a-Judge correlates more closely with human judgments than multiple-choice metrics, and that informativeness plays a critical role in truthfulness assessment. Our results also indicate that machine translation provides a viable approach for extending truthfulness benchmarks to additional languages, offering a scalable alternative to professional translation. Finally, we observe that universal knowledge questions are better handled across languages than context- and time-dependent ones, highlighting the need for truthfulness evaluations that account for cultural and temporal variability. Dataset and code are publicly available under open licenses.
+
+### Citation
+
+```text
+@misc{figueras2025truthknowslanguageevaluating,
+      title={Truth Knows No Language: Evaluating Truthfulness Beyond English},
+      author={Blanca Calvo Figueras and Eneko Sagarzazu and Julen Etxaniz and Jeremy Barnes and Pablo Gamallo and Iria De Dios Flores and Rodrigo Agerri},
+      year={2025},
+      eprint={2502.09387},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2502.09387},
+}
+```
+
+### Groups, Tags, and Tasks
+
+#### Groups
+
+* `truthfulqa`: This task follows the [TruthfulQA dataset](https://arxiv.org/abs/2109.07958), but expands it to new languages.
+
+#### Tasks
+
+* `truthfulqa-multi_mc2_es`: `Multiple-choice, multiple answers in Spanish`
+* `truthfulqa-multi_gen_es`: `Answer generation in Spanish`
+* `truthfulqa-multi_mc2_ca`: `Multiple-choice, multiple answers in Catalan`
+* `truthfulqa-multi_gen_ca`: `Answer generation in Catalan`
+* `truthfulqa-multi_mc2_eu`: `Multiple-choice, multiple answers in Basque`
+* `truthfulqa-multi_gen_eu`: `Answer generation in Basque`
+* `truthfulqa-multi_mc2_gl`: `Multiple-choice, multiple answers in Galician`
+* `truthfulqa-multi_gen_gl`: `Answer generation in Galician`
+* `truthfulqa-multi_mc2_en`: `Multiple-choice, multiple answers in English`
+* `truthfulqa-multi_gen_en`: `Answer generation in English`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+
+* [X] Is the task an existing benchmark in the literature?
+  * [X] Have you referenced the original paper that introduced the task?
+  * [X] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+If other tasks on this dataset are already supported:
+
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?
+
+### Changelog

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_ca.yaml

+include: truthfulqa-multi_gen_common
+task: truthfulqa-multi_gen_ca
+dataset_name: ca

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_common

+tag:
+  - truthfulqa_multi
+dataset_path: HiTZ/truthfulqa-multi
+output_type: generate_until
+generation_kwargs:
+  until:
+    - "!\n\n"
+    - "Q:"
+    - ".\n\n"
+training_split: train
+validation_split: validation
+test_split: null
+doc_to_target: "{{'A: ' + best_answer}}"
+fewshot_split: train
+fewshot_config:
+  sampler: first_n
+process_docs: !function utils.process_docs_gen
+process_results: !function utils.process_results_gen
+doc_to_text: "{{'Q: ' + question}}"
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  # - metric: bleurt_max
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: bleurt_acc
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: bleurt_diff
+  #   aggregation: mean
+  #   higher_is_better: true
+  - metric: bleu_max
+    aggregation: mean
+    higher_is_better: true
+  - metric: bleu_acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: bleu_diff
+    aggregation: mean
+    higher_is_better: true
+  #- metric: rouge1_max
+  #  aggregation: mean
+  #  higher_is_better: true
+  #- metric: rouge1_acc
+  #  aggregation: mean
+  #   higher_is_better: true
+  # - metric: rouge1_diff
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rouge2_max
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rouge2_acc
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rouge2_diff
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rougeL_max
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rougeL_acc
+  #   aggregation: mean
+  #   higher_is_better: true
+  # - metric: rougeL_diff
+  #   aggregation: mean
+  #   higher_is_better: true
+metadata:
+  version: 3.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_en.yaml

+include: truthfulqa-multi_gen_common
+task: truthfulqa-multi_gen_en
+dataset_name: en

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_es.yaml

+include: truthfulqa-multi_gen_common
+task: truthfulqa-multi_gen_es
+dataset_name: es

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_eu.yaml

+include: truthfulqa-multi_gen_common
+task: truthfulqa-multi_gen_eu
+dataset_name: eu

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_gen_gl.yaml

+include: truthfulqa-multi_gen_common
+task: truthfulqa-multi_gen_gl
+dataset_name: gl

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc1_ca.yaml

+include: truthfulqa-multi_mc_common
+task: truthfulqa-multi_mc1_ca
+dataset_name: ca

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc1_en.yaml

+include: truthfulqa-multi_mc_common
+task: truthfulqa-multi_mc1_en
+dataset_name: en

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc1_es.yaml

+include: truthfulqa-multi_mc_common
+task: truthfulqa-multi_mc1_es
+dataset_name: es

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc1_eu.yaml

+include: truthfulqa-multi_mc_common
+task: truthfulqa-multi_mc1_eu
+dataset_name: eu

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc1_gl.yaml

+include: truthfulqa-multi_mc_common
+task: truthfulqa-multi_mc1_gl
+dataset_name: gl

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc2_ca.yaml

+include: truthfulqa-multi_mc1_ca.yaml
+task: truthfulqa-multi_mc2_ca
+doc_to_choice: "{{mc2_targets.choices}}"
+process_results: !function utils.process_results_mc2
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc2_en.yaml

+include: truthfulqa-multi_mc1_en.yaml
+task: truthfulqa-multi_mc2_en
+doc_to_choice: "{{mc2_targets.choices}}"
+process_results: !function utils.process_results_mc2
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc2_es.yaml

+include: truthfulqa-multi_mc1_es.yaml
+task: truthfulqa-multi_mc2_es
+doc_to_choice: "{{mc2_targets.choices}}"
+process_results: !function utils.process_results_mc2
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc2_eu.yaml

+include: truthfulqa-multi_mc1_eu.yaml
+task: truthfulqa-multi_mc2_eu
+doc_to_choice: "{{mc2_targets.choices}}"
+process_results: !function utils.process_results_mc2
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc2_gl.yaml

+include: truthfulqa-multi_mc1_gl.yaml
+task: truthfulqa-multi_mc2_gl
+doc_to_choice: "{{mc2_targets.choices}}"
+process_results: !function utils.process_results_mc2
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0

lm_eval/tasks/truthfulqa-multi/truthfulqa-multi_mc_common

+tag:
+  - truthfulqa-multi
+dataset_path: HiTZ/truthfulqa-multi
+output_type: multiple_choice
+training_split: train
+validation_split: validation
+test_split: null
+fewshot_split: train
+fewshot_config:
+  sampler: first_n
+doc_to_target: 0
+doc_to_choice: "{{mc1_targets.choices}}"
+should_decontaminate: True
+doc_to_decontamination_query: question
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 2.0
+doc_to_text: "{{'Q: ' + question + '\nA:'}}"

lm_eval/tasks/truthfulqa-multi/utils.py

+import logging
+
+import datasets
+import numpy as np
+
+
+logger = logging.getLogger(__name__)
+
+try:
+    import sacrebleu
+    from rouge_score import rouge_scorer, scoring
+except ImportError as e:
+    raise type(e)(
+        "Required packages not installed. Please install the required packages via `pip install rouge_score sacrebleu`"
+    )
+
+
+def process_results_mc2(doc, results):
+    lls, is_greedy = zip(*results)
+
+    # Split on the first `0` as everything before it is true (`1`).
+    split_idx = list(doc["mc2_targets"]["labels"]).index(0)
+    # Compute the normalized probability mass for the correct answer.
+    ll_true, ll_false = lls[:split_idx], lls[split_idx:]
+    p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))
+    p_true = p_true / (sum(p_true) + sum(p_false))
+    accuracy = sum(p_true)
+    # TODO. alternatively
+    # new = (sum(p_true)/p_true) / ((sum(p_true)/p_true) + (sum(p_false)/p_false))
+
+    return {"acc": accuracy}  # , "mc2_new":new}
+
+
+def process_docs_gen(dataset: datasets.Dataset) -> datasets.Dataset:
+    return dataset.map(preprocess_function)
+
+
+def preprocess_function(examples):
+    def _format_answers(answers):
+        formatted_answers = []
+        for answer in answers:
+            answer = answer.strip()
+            if len(answer):
+                # Add a period after all answers.
+                if answer[-1] != ".":
+                    formatted_answers.append(answer + ".")
+                else:
+                    formatted_answers.append(answer)
+        return formatted_answers
+
+    if "correct_answers" in examples:
+        incorrect_answers = _format_answers(examples["incorrect_answers"])
+        correct_answers = _format_answers(examples["correct_answers"])
+        if examples["lang"] == "en" and "I have no comment." not in correct_answers:
+            correct_answers.append("I have no comment.")
+        elif (
+            examples["lang"] == "ca" and "No tinc cap comentari." not in correct_answers
+        ):
+            correct_answers.append("No tinc cap comentari.")
+        elif (
+            examples["lang"] == "es"
+            and "No tengo ningún comentario." not in correct_answers
+        ):
+            correct_answers.append("No tengo ningún comentario.")
+        elif examples["lang"] == "eu" and "Iruzkinik ez." not in correct_answers:
+            correct_answers.append("Iruzkinik ez.")
+        elif (
+            examples["lang"] == "gl"
+            and "Non teño ningún comentario." not in correct_answers
+        ):
+            correct_answers.append("Non teño ningún comentario.")
+    return {
+        "question": examples["question"].strip(),
+        "correct_answers": correct_answers,
+        "incorrect_answers": incorrect_answers,
+        "best_answer": examples["best_answer"],
+    }
+
+
+def process_results_gen(doc, results):
+    completion = results[0]
+    true_refs, false_refs = doc["correct_answers"], doc["incorrect_answers"]
+    all_refs = true_refs + false_refs
+
+    # Process the sentence-level BLEURT, BLEU, and ROUGE for similarity measures.
+
+    # # BLEURT
+    # bleurt_scores_true = self.bleurt.compute(
+    #     predictions=[completion] * len(true_refs), references=true_refs
+    # )["scores"]
+    # bleurt_scores_false = self.bleurt.compute(
+    #     predictions=[completion] * len(false_refs), references=false_refs
+    # )["scores"]
+    # bleurt_correct = max(bleurt_scores_true)
+    # bleurt_incorrect = max(bleurt_scores_false)
+    # bleurt_max = bleurt_correct
+    # bleurt_diff = bleurt_correct - bleurt_incorrect
+    # bleurt_acc = int(bleurt_correct > bleurt_incorrect)
+
+    # BLEU
+    bleu_scores = [bleu([[ref]], [completion]) for ref in all_refs]
+    bleu_correct = np.nanmax(bleu_scores[: len(true_refs)])
+    bleu_incorrect = np.nanmax(bleu_scores[len(true_refs) :])
+    bleu_max = bleu_correct
+    bleu_diff = bleu_correct - bleu_incorrect
+    bleu_acc = int(bleu_correct > bleu_incorrect)
+
+    # ROUGE-N
+    # rouge_scores = [rouge([ref], [completion]) for ref in all_refs]
+    # # ROUGE-1
+    # rouge1_scores = [score["rouge1"] for score in rouge_scores]
+    # rouge1_correct = np.nanmax(rouge1_scores[: len(true_refs)])
+    # rouge1_incorrect = np.nanmax(rouge1_scores[len(true_refs) :])
+    # rouge1_max = rouge1_correct
+    # rouge1_diff = rouge1_correct - rouge1_incorrect
+    # rouge1_acc = int(rouge1_correct > rouge1_incorrect)
+    # # ROUGE-2
+    # rouge2_scores = [score["rouge2"] for score in rouge_scores]
+    # rouge2_correct = np.nanmax(rouge2_scores[: len(true_refs)])
+    # rouge2_incorrect = np.nanmax(rouge2_scores[len(true_refs) :])
+    # rouge2_max = rouge2_correct
+    # rouge2_diff = rouge2_correct - rouge2_incorrect
+    # rouge2_acc = int(rouge2_correct > rouge2_incorrect)
+    # # ROUGE-L
+    # rougeL_scores = [score["rougeLsum"] for score in rouge_scores]
+    # rougeL_correct = np.nanmax(rougeL_scores[: len(true_refs)])
+    # rougeL_incorrect = np.nanmax(rougeL_scores[len(true_refs) :])
+    # rougeL_max = rougeL_correct
+    # rougeL_diff = rougeL_correct - rougeL_incorrect
+    # rougeL_acc = int(rougeL_correct > rougeL_incorrect)
+
+    return {
+        # "bleurt_max": bleurt_max,
+        # "bleurt_acc": bleurt_acc,
+        # "bleurt_diff": bleurt_diff,
+        "bleu_max": bleu_max,
+        "bleu_acc": bleu_acc,
+        "bleu_diff": bleu_diff,
+        # "rouge1_max": rouge1_max,
+        # "rouge1_acc": rouge1_acc,
+        # "rouge1_diff": rouge1_diff,
+        # "rouge2_max": rouge2_max,
+        # "rouge2_acc": rouge2_acc,
+        # "rouge2_diff": rouge2_diff,
+        # "rougeL_max": rougeL_max,
+        # "rougeL_acc": rougeL_acc,
+        # "rougeL_diff": rougeL_diff,
+    }
+
+
+def bleu(refs, preds):
+    """
+    Returns `t5` style BLEU scores. See the related implementation:
+    https://github.com/google-research/text-to-text-transfer-transformer/blob/3d10afd51ba97ac29eb66ae701eca274488202f7/t5/evaluation/metrics.py#L41
+
+    :param refs:
+        A `list` of `list` of reference `str`s.
+    :param preds:
+        A `list` of predicted `str`s.
+    """
+    score = sacrebleu.corpus_bleu(
+        preds,
+        refs,
+        smooth_method="exp",
+        smooth_value=0.0,
+        force=False,
+        lowercase=False,
+        tokenize="intl",
+        use_effective_order=False,
+    ).score
+    return score
+
+
+def rouge(refs, preds):
+    """
+    Returns `t5` style ROUGE scores. See the related implementation:
+    https://github.com/google-research/text-to-text-transfer-transformer/blob/3d10afd51ba97ac29eb66ae701eca274488202f7/t5/evaluation/metrics.py#L68
+
+    :param refs:
+        A `list` of reference `strs`.
+    :param preds:
+        A `list` of predicted `strs`.
+    """
+    rouge_types = ["rouge1", "rouge2", "rougeLsum"]
+    scorer = rouge_scorer.RougeScorer(rouge_types)
+    # Add newlines between sentences to correctly compute `rougeLsum`.
+
+    def _prepare_summary(summary):
+        summary = summary.replace(" . ", ".\n")
+        return summary
+
+    # Accumulate confidence intervals.
+    aggregator = scoring.BootstrapAggregator()
+    for ref, pred in zip(refs, preds):
+        ref = _prepare_summary(ref)
+        pred = _prepare_summary(pred)
+        aggregator.add_scores(scorer.score(ref, pred))
+    result = aggregator.aggregate()
+    return {type: result[type].mid.fmeasure * 100 for type in rouge_types}