Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
ruin_names.yaml 1.08 KB
Newer Older
Nathan Habib's avatar
Adds Open LLM Leaderboard Taks (#2047)  
Nathan Habib committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50 
dataset_name: ruin_names
description: 'Select the humorous edit that ''ruins'' the input movie or musical artist
  name.

        '
doc_to_choice: ["(A)","(B)","(C)","(D)","(E)","(F)"]
fewshot_config:
  sampler: first_n
  samples:
  - input: 'Which of the following is a humorous edit of this artist or movie name:
      ''whitesnake''?

      Options:

      (A) whitesnape

      (B) whitesnapke

      (C) whitesnuake

      (D) mwhitesnake'
    target: (A)
  - input: 'Which of the following is a humorous edit of this artist or movie name:
      ''one of our dinosaurs is missing''?

      Options:

      (A) ofne of our dinosaurs is missing

      (B) one af our dinosaurs is missing

      (C) one of our dinosaurs is pissing

      (D) one of our dinosaur is missing'
    target: (C)
  - input: 'Which of the following is a humorous edit of this artist or movie name:
      ''counting crows''?

      Options:

      (A) countingy crows

      (B) counting cows

      (C) courting crows

      (D) coutnting crows'
    target: (B)
include: _fewshot_template_yaml
task: leaderboard_bbh_ruin_names