Implement NoticIA (#1912)

* Noticia

* test

* Final testes implementation

* Fixes

* Fix linters

lm_eval/tasks/noticia/README.md

+# NoticIA
+
+### Paper
+
+Title: `NoticIA: A Clickbait Article Summarization Dataset in Spanish`
+
+Abstract: https://arxiv.org/abs/2404.07611
+
+We present NoticIA, a dataset consisting of 850 Spanish news articles featuring prominent clickbait headlines, each paired with high-quality, single-sentence generative summarizations written by humans. This task demands advanced text understanding and summarization abilities, challenging the models' capacity to infer and connect diverse pieces of information to meet the user's informational needs generated by the clickbait headline. We evaluate the Spanish text comprehension capabilities of a wide range of state-of-the-art large language models. Additionally, we use the dataset to train ClickbaitFighter, a task-specific model that achieves near-human performance in this task.
+
+Homepage: https://github.com/ikergarcia1996/NoticIA
+
+### Citation
+
+```
+@article{noticia2024,
+      title={NoticIA: A Clickbait Article Summarization Dataset in Spanish},
+      author={Iker García-Ferrero and Begoña Altuna},
+      year={2024},
+      journal = {Procesamiento del Lenguaje Natural},
+      volume = {73},
+      number = {0},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* Not part of a group yet.
+
+#### Tasks
+
+* `noticia`
+
+#### Metrics
+
+Following the original implementation, this task will compute the 'Rouge1 score' and 'Average Summary Length.'
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [x] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [x] Have you noted which, if any, published evaluation setups are matched by this variant?

lm_eval/tasks/noticia/noticia.yaml

+task: noticia
+dataset_path: Iker/NoticIA
+dataset_name: null
+output_type: generate_until
+generation_kwargs:
+  until:
+    - "\n\n"
+    - "\n"
+  do_sample: false
+  temperature: 0.0
+training_split: null
+validation_split: null
+test_split: test
+fewshot_split: null
+doc_to_text: "Ahora eres una Inteligencia Artificial experta en desmontar titulares sensacionalistas o clickbait. Tu tarea consiste en analizar noticias con titulares sensacionalistas y generar un resumen de una sola frase que revele la verdad detrás del titular.\nEste es el titular de la noticia: {{web_headline}}\nEl titular plantea una pregunta o proporciona información incompleta. Debes buscar en el cuerpo de la noticia una frase que responda lo que se sugiere en el título. Responde siempre que puedas parafraseando el texto original. Usa siempre las mínimas palabras posibles. Recuerda responder siempre en Español.\nEste es el cuerpo de la noticia:\n{{web_text}}"
+doc_to_target: summary
+target_delimiter: " "
+num_fewshot: 0
+should_decontaminate: false
+doc_to_decontamination_query: sentence
+metric_list:
+  - metric: !function utils.rouge1
+    higher_is_better: true
+    aggregation: !function utils.rouge1_agg
+  - metric: !function utils.average_len
+    higher_is_better: false
+    aggregation: !function utils.average_len_agg
+metadata:
+  version: 1.0

lm_eval/tasks/noticia/utils.py

+import string
+
+import evaluate
+
+
+def clean_text(text: str) -> str:
+    # Remove punctuation
+    text = text.translate(str.maketrans("", "", string.punctuation))
+
+    # Remove newlines and multiple spaces
+    text = text.replace("\n", " ").strip()
+    text = " ".join(text.split()).strip()
+
+    # lowercase
+    text = text.lower()
+
+    return text
+
+
+def rouge1(items):
+    """
+    # passthrough for efficiency
+    """
+    return items
+
+
+def average_len(items):
+    """
+    # passthrough for efficiency
+    """
+    return items
+
+
+def rouge1_agg(items):
+    """
+    Higher is better
+    """
+
+    refs = list(zip(*items))[0]
+    refs = [[clean_text(ref)] for ref in refs]
+    # print("refs", refs)
+    preds = [clean_text(x) for x in list(zip(*items))[1]]
+    # print("preds", preds)
+    rouge_scorer = evaluate.load("rouge")
+    return rouge_scorer.compute(predictions=preds, references=refs)["rouge1"]
+
+
+def average_len_agg(items):
+    """
+    Higher is better
+    """
+
+    preds = [clean_text(x) for x in list(zip(*items))[1]]
+
+    return sum(len(x.split()) for x in preds) / len(preds)