[Feature] update drop dataset from openai simple eval (#1092)

* [Feature] update drop dataset from openai simple eval

* update drop template presentation

* update

---------
Co-authored-by: Leymore <zfz-960727@163.com>

configs/datasets/drop/drop_gen.py

 from mmengine.config import read_base
 
 with read_base():
-    from .drop_gen_8a9ed9 import drop_datasets  # noqa: F401, F403
+    from .drop_gen_822d33 import drop_datasets

configs/datasets/drop/drop_gen_e18aaf.py

+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import DropOpenAIDataset, DropOpenAIEvaluator
+
+drop_reader_cfg = dict(
+    input_columns=['prompt'],
+    output_column='answers',
+    train_split='validation',
+    test_split='validation',
+)
+
+template = '''\
+You will be asked to read a passage and answer a question. Some examples of passages and Q&A are provided below.
+
+# Examples
+---
+Passage: Trunajaya rebellion  or Trunajaya War was the ultimately unsuccessful rebellion waged by the Madurese prince Trunajaya and fighters from Makassar against the Mataram Sultanate and its Dutch East India Company  supporters in Java  during the 1670s. The rebellion was initially successful: the rebels defeated the royal army at Gegodog , captured most of the Javanese north coast, and took the Mataram capital Plered . King Amangkurat I died during the retreat of the royal court. His son and successor, Amangkurat II, requested help from the VOC in exchange for financial remuneration and geopolitical concessions. The VOC\'s subsequent involvement turned the tide of the war. VOC and Mataram forces recovered lost territories and overran Trunajaya\'s new capital at Kediri . However, the rebellion continued until the capture of Trunajaya at the end of 1679, and the defeat, death, or surrender of the other rebel leaders . Trunajaya was killed by Amangkurat II personally in 1680 while a prisoner of the VOC. After his father\'s death in 1677, Amangkurat II also faced rival claims to the throne. The most serious rival was his brother Pangeran Puger, who took the capital Plered in 1677 and did not surrender until 1681.
+Question: How many years was it between Trunajaya\'s capture and his death while prisoner of the VOC?
+Answer:  1
+
+---
+Passage: Led by former Giant Kurt Warner, the defending NFC champions took the field at Giants Stadium against a Giants team still reeling from their bad loss in New Orleans. The Giants scored first, sending Jacobs in for a 4-yard touchdown run following a Terrell Thomas interception. Later, Arizona running back Beanie Wells scored his first career touchdown on a 13-yard rush. Manning responded by throwing a 62-yard touchdown to Nicks for his longest reception of the year. In the second half, the Cardinals\' Tim Hightower and Jason Wright scored touchdowns. But it was turnovers that decided this game; Manning\'s 3 interceptions were as many as he had thrown all season. The Giants scored only 3 points in the second half, ending the game on an interception to Antrel Rolle. The Giants notable streak of 38 consecutive starts by the same offensive line unit was ended here, as offensive tackle Kareem McKenzie missed the game with a groin injury. McKenzie returned the following week.
+Question: Which player made the first score of the game?
+Answer:  Jacobs
+
+---
+Passage: Hoping to rebound from their road loss to the Bills, the Chargers flew to Wembley Stadium for the 2008 International Series game with the New Orleans Saints. In the first quarter, San Diego trailed early as kicker Taylor Mehlhaff got a 23-yard field goal.  The \'Bolts would respond with kicker Nate Kaeding getting a 33-yard field goal.  In the second quarter, New Orleans regained the lead as QB Drew Brees (a former Charger) completed a 12-yard TD pass to WR Devery Henderson (with a failed PAT) and RB Deuce McAllister getting a 1-yard TD run.  San Diego answered as QB Philip Rivers completed a 12-yard TD pass to RB LaDainian Tomlinson, but the Saints replied with Brees completing a 30-yard TD pass to WR Lance Moore.  The Chargers closed out the half with Rivers completing a 12-yard TD pass to TE Antonio Gates. In the third quarter, New Orleans increased its lead Brees completing a 1-yard TD pass to TE Mark Campbell, after a very controversial Pass interference call on cornerback Cletis Gordon put the Saints on the 1-yard line.  The \'Bolts would answer with Kaeding getting a 24-yard field goal.  In the fourth quarter, the Saints continued to build its lead as FB Mike Karney got a 1-yard TD run.  San Diego tried to rally as Kaeding nailed a 31-yard field goal, Rivers completed a 14-yard TD pass to WR Vincent Jackson, and Brees giving the \'Bolts a safety via an incomplete pass thrown into the back of his own endzone.  However, New Orleans\' defense stiffened for the win. With the loss, the Chargers went into their bye week at 3-5.
+Question: How many total yards of touchdown passes did Drew Brees make?
+Answer:  43
+
+# Your Task
+
+---
+{prompt}
+
+Think step by step, then write a line of the form "Answer: $ANSWER" at the end of your response.
+'''
+
+drop_infer_cfg = dict(
+    prompt_template=dict(type=PromptTemplate, template=template),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+drop_eval_cfg = dict(evaluator=dict(type=DropOpenAIEvaluator))
+
+drop_datasets = [
+    dict(
+        abbr='drop',
+        type=DropOpenAIDataset,
+        path='./data/drop_simple_eval/dev.jsonl',
+        reader_cfg=drop_reader_cfg,
+        infer_cfg=drop_infer_cfg,
+        eval_cfg=drop_eval_cfg)
+]

opencompass/datasets/__init__.py

@@ -33,6 +33,7 @@ from .custom import *  # noqa: F401, F403
 from .cvalues import *  # noqa: F401, F403
 from .drcd import *  # noqa: F401, F403
 from .drop import *  # noqa: F401, F403
+from .drop_simple_eval import *  # noqa: F401, F403
 from .ds1000 import *  # noqa: F401, F403
 from .ds1000_interpreter import *  # noqa: F401, F403
 from .eprstmt import *  # noqa: F401, F403

opencompass/datasets/drop_simple_eval.py

+import json
+import re
+import string
+
+from datasets import Dataset, DatasetDict
+
+from opencompass.openicl.icl_evaluator import BaseEvaluator
+from opencompass.registry import LOAD_DATASET
+
+from .base import BaseDataset
+
+# Modified from https://github.com/openai/simple-evals/blob/main/drop_eval.py
+
+ANSWER_PATTERN = r'(?i)Answer\s*:\s*([^\n]+)'
+
+
+def normalize(s: str) -> str:
+    """Lower text and remove punctuation, articles and extra whitespace."""
+    s = s.lower()
+    exclude = set(string.punctuation)
+    s = ''.join(char for char in s if char not in exclude)
+    s = re.sub(r'\b(a|an|the)\b', ' ', s)
+    s = ' '.join(s.split())
+    return s
+
+
+def fuzzy_match(s1: str, s2: str) -> bool:
+    s1 = normalize(s1)
+    s2 = normalize(s2)
+
+    if s1 == '' or s2 == '':
+        return s1 == s2
+
+    return s1 in s2 or s2 in s1
+
+
+@LOAD_DATASET.register_module()
+class DropOpenAIDataset(BaseDataset):
+
+    @staticmethod
+    def load(path):
+        dataset_list = []
+        with open(path, 'r', encoding='utf-8') as f:
+            for line in f:
+                data = json.loads(line)
+                item = {
+                    'prompt': data['context'],
+                    'answers': data['ref_text'],
+                }
+                dataset_list.append(item)
+
+        dataset_list = Dataset.from_list(dataset_list)
+        return DatasetDict({'validation': dataset_list})
+
+
+class DropOpenAIEvaluator(BaseEvaluator):
+
+    def score(self, predictions, references):
+        if len(predictions) != len(references):
+            return {'error': 'preds and refers have different length'}
+        num_correct = 0
+        count = 0
+        details = []
+        for pred, refr in zip(predictions, references):
+            match = re.search(ANSWER_PATTERN, pred)
+            extracted_answer = match.group(1) if match else pred
+            refrs = refr.split('|')
+            matches = [
+                fuzzy_match(extracted_answer, correct_answer)
+                for correct_answer in refrs
+            ]
+            correct = True in matches
+            num_correct += correct
+
+            detail = {'pred': pred, 'answer': refr, 'correct': correct}
+            count += 1
+
+            details.append(detail)
+        result = {'accuracy': 100 * num_correct / count, 'details': details}
+        return result