Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
snarks.yaml 1.36 KB
Newer Older
Nathan Habib's avatar
Adds Open LLM Leaderboard Taks (#2047)  
Nathan Habib committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45 
dataset_name: snarks
description: 'Determine which of two sentences is sarcastic.


  According to Cambridge University Dictionary, sarcasm is "the use of remarks that
  clearly mean the opposite of what they say, made in order to hurt someone''s feelings
  or to criticize something in a humorous way." Sarcastic sentences often contain
  satirical or ironic utterances, hyperboles, ambivalent or witty remarks.


  '
doc_to_choice: ["(A)","(B)"]
fewshot_config:
  sampler: first_n
  samples:
  - input: 'Which statement is sarcastic?

      Options:

      (A) Yes, because having interests and actively researching them is a huge waste

      (B) Yes, because having interests and actively researching them is a huge deal'
    target: (A)
  - input: 'Which statement is sarcastic?

      Options:

      (A) No one is going to disagree with you on this. Avoiding ad hominem attacks
      really help your case

      (B) No one is going to disagree with you on this. Ad hominem attacks really
      help your case'
    target: (B)
  - input: 'Which statement is sarcastic?

      Options:

      (A) Consistency in the league''s punishments? What do you think this is supposed
      to be, politics?

      (B) Consistency in the league''s punishments? What do you think this is supposed
      to be, moral?'
    target: (A)
include: _fewshot_template_yaml
task: leaderboard_bbh_snarks