Merge branch 'EleutherAI:main' into main

lm_eval/tasks/french_bench/french_bench_orangesum_abstract.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_gen
 description: "Résume l'article en une phrase."

lm_eval/tasks/french_bench/french_bench_orangesum_title.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_extra
 description: "Trouve le titre de l'article."

lm_eval/tasks/french_bench/french_bench_reading_comp.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_extra
 # description: "Répond au mieux en complétant la question avec une des réponses proposées."

lm_eval/tasks/french_bench/french_bench_topic_based_nli.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_extra
 description: "A propos du thème spécifié, l'avis client est il positif, négatif, ou neutre ?"

lm_eval/tasks/french_bench/french_bench_trivia.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_gen
 task: french_bench_trivia

lm_eval/tasks/french_bench/french_bench_vocab.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_mc
 # description: "Répond au mieux en complétant la question avec une des réponses proposées."

lm_eval/tasks/french_bench/french_bench_wikitext_fr.yaml

-group:
+tag:
   - french_bench_perplexity
 task: french_bench_wikitext_fr
 dataset_path: asi/wikitext_fr

lm_eval/tasks/french_bench/french_bench_xnli.yaml

 include: "_default_template_yaml"
-group:
+tag:
   - french_bench
   - french_bench_extra
 description: "La prémisse et l'hypothèse sont elles en accord, neutres en elles, ou en contradiction ?"

lm_eval/tasks/glianorex/README.md

+# Glianorex
+
+The goal of this benchmark is to isolate the test answering capabilities from the content knowledge.
+
+### Paper
+
+Title: Multiple Choice Questions and Large Languages Models: A Case Study with Fictional Medical Data
+
+Abstract: https://arxiv.org/abs/2406.02394
+
+To test the relevance of MCQs to assess LLM performance without prior data exposure, we created a fictional medical benchmark and knowledge base on a non-existent gland, the Glianorex. Using GPT-4 we generated a comprehensive textbook on the Glianorex in both English and French, and created multiple-choice questions in both English and French.
+
+### Tasks
+
+All tasks are multiple choice questions with 4 options, only one correct option.
+
+- `glianorex`: Evaluates all tasks listed below.
+
+- `glianorex_en`: Evaluates the accuracy on 264 questions in English.
+- `glianorex_fr`: Evaluates the accuracy on 264 questions in French.

lm_eval/tasks/glianorex/glianorex.yaml

+task: glianorex
+dataset_path: maximegmd/glianorex
+output_type: multiple_choice
+test_split: train
+doc_to_text: !function preprocess_glianorex.doc_to_text
+doc_to_target: !function preprocess_glianorex.doc_to_target
+doc_to_choice: [ 'A', 'B', 'C', 'D' ]
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true

lm_eval/tasks/glianorex/glianorex_en.yaml

+task: glianorex_en
+dataset_path: maximegmd/glianorex
+output_type: multiple_choice
+test_split: train
+doc_to_text: !function preprocess_glianorex.doc_to_text
+doc_to_target: !function preprocess_glianorex.doc_to_target
+process_docs: !function preprocess_glianorex.filter_english
+doc_to_choice: [ 'A', 'B', 'C', 'D' ]
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true

lm_eval/tasks/glianorex/glianorex_fr.yaml

+task: glianorex_fr
+dataset_path: maximegmd/glianorex
+output_type: multiple_choice
+test_split: train
+doc_to_text: !function preprocess_glianorex.doc_to_text
+doc_to_target: !function preprocess_glianorex.doc_to_target
+process_docs: !function preprocess_glianorex.filter_french
+doc_to_choice: [ 'A', 'B', 'C', 'D' ]
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true

lm_eval/tasks/glianorex/preprocess_glianorex.py

+import datasets
+
+
+def doc_to_text(doc) -> str:
+    option_choices = doc["options"]
+    answers = "".join((f"{k}. {v}\n") for k, v in option_choices.items())
+    return f"Question: {doc['question']}\n{answers}Answer:"
+
+
+def doc_to_target(doc) -> int:
+    return doc["answer_idx"]
+
+
+def filter_dataset(dataset: datasets.Dataset, lang: str) -> datasets.Dataset:
+    return dataset.filter(lambda example: example["language"].startswith(lang))
+
+
+def filter_french(dataset: datasets.Dataset) -> datasets.Dataset:
+    return filter_dataset(dataset, "fr")
+
+
+def filter_english(dataset: datasets.Dataset) -> datasets.Dataset:
+    return filter_dataset(dataset, "en")

lm_eval/tasks/glue/README.md

@@ -41,10 +41,14 @@ Homepage: https://gluebenchmark.com/
 }
 ```
 
-### Groups and Tasks
+### Groups, Tags, and Tasks
 
 #### Groups
 
+None.
+
+#### Tags
+
 * `glue`: Run all Glue subtasks.
 
 #### Tasks

lm_eval/tasks/glue/cola/default.yaml

-group: glue
+tag: glue
 task: cola
 dataset_path: glue
 dataset_name: cola

lm_eval/tasks/glue/mnli/default.yaml

-group: glue
+tag: glue
 task: mnli
 dataset_path: glue
 dataset_name: mnli

lm_eval/tasks/glue/mrpc/default.yaml

-group: glue
+tag: glue
 task: mrpc
 dataset_path: glue
 dataset_name: mrpc

lm_eval/tasks/glue/qnli/default.yaml

-group: glue
+tag: glue
 task: qnli
 dataset_path: glue
 dataset_name: qnli

lm_eval/tasks/glue/qqp/default.yaml

-group: glue
+tag: glue
 task: qqp
 dataset_path: glue
 dataset_name: qqp

lm_eval/tasks/glue/rte/default.yaml

-group: glue
+tag: glue
 task: rte
 dataset_path: glue
 dataset_name: rte