Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
97e3c9fe
Commit
97e3c9fe
authored
Apr 18, 2024
by
lintangsutawika
Browse files
added lama primed and negated dataset
parent
7852985b
Changes
61
Show whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
122 additions
and
0 deletions
+122
-0
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_google_re.yaml
.../lama_primed_negated/right_wrong/misprimed_google_re.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_squad.yaml
...asks/lama_primed_negated/right_wrong/misprimed_squad.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_trex.yaml
...tasks/lama_primed_negated/right_wrong/misprimed_trex.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/negated_conceptnet.yaml
...s/lama_primed_negated/right_wrong/negated_conceptnet.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/negated_google_re.yaml
...ks/lama_primed_negated/right_wrong/negated_google_re.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/negated_squad.yaml
.../tasks/lama_primed_negated/right_wrong/negated_squad.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/negated_trex.yaml
...l/tasks/lama_primed_negated/right_wrong/negated_trex.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/normal_conceptnet.yaml
...ks/lama_primed_negated/right_wrong/normal_conceptnet.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/normal_google_re.yaml
...sks/lama_primed_negated/right_wrong/normal_google_re.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/normal_squad.yaml
...l/tasks/lama_primed_negated/right_wrong/normal_squad.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/right_wrong/normal_trex.yaml
...al/tasks/lama_primed_negated/right_wrong/normal_trex.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/_misprimed_true_false_template_yaml
...ed_negated/true_false/_misprimed_true_false_template_yaml
+18
-0
lm_eval/tasks/lama_primed_negated/true_false/_negated_true_false_template_yaml
...imed_negated/true_false/_negated_true_false_template_yaml
+18
-0
lm_eval/tasks/lama_primed_negated/true_false/_normal_true_false_template_yaml
...rimed_negated/true_false/_normal_true_false_template_yaml
+18
-0
lm_eval/tasks/lama_primed_negated/true_false/misprimed_conceptnet.yaml
.../lama_primed_negated/true_false/misprimed_conceptnet.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/misprimed_google_re.yaml
...s/lama_primed_negated/true_false/misprimed_google_re.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/misprimed_squad.yaml
...tasks/lama_primed_negated/true_false/misprimed_squad.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/misprimed_trex.yaml
.../tasks/lama_primed_negated/true_false/misprimed_trex.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/negated_conceptnet.yaml
...ks/lama_primed_negated/true_false/negated_conceptnet.yaml
+4
-0
lm_eval/tasks/lama_primed_negated/true_false/negated_google_re.yaml
...sks/lama_primed_negated/true_false/negated_google_re.yaml
+4
-0
No files found.
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_google_re.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_right_wrong_template_yaml
task
:
lama_misprimed_google_re_right_wrong
dataset_name
:
GoogleRE
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_squad.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_right_wrong_template_yaml
task
:
lama_misprimed_squad_right_wrong
dataset_name
:
SQUAD
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/misprimed_trex.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_right_wrong_template_yaml
task
:
lama_misprimed_trex_right_wrong
dataset_name
:
TREx
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/negated_conceptnet.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_right_wrong_template_yaml
task
:
lama_negated_conceptnet_right_wrong
dataset_name
:
ConceptNet
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/negated_google_re.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_right_wrong_template_yaml
task
:
lama_negated_google_re_right_wrong
dataset_name
:
GoogleRE
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/negated_squad.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_right_wrong_template_yaml
task
:
lama_negated_squad_right_wrong
dataset_name
:
SQUAD
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/negated_trex.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_right_wrong_template_yaml
task
:
lama_negated_trex_right_wrong
dataset_name
:
TREx
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/normal_conceptnet.yaml
0 → 100755
View file @
97e3c9fe
include
:
_normal_right_wrong_template_yaml
task
:
lama_normal_conceptnet_right_wrong
dataset_name
:
ConceptNet
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/normal_google_re.yaml
0 → 100755
View file @
97e3c9fe
include
:
_normal_right_wrong_template_yaml
task
:
lama_normal_google_re_right_wrong
dataset_name
:
GoogleRE
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/normal_squad.yaml
0 → 100755
View file @
97e3c9fe
include
:
_normal_right_wrong_template_yaml
task
:
lama_normal_squad_right_wrong
dataset_name
:
SQUAD
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/right_wrong/normal_trex.yaml
0 → 100755
View file @
97e3c9fe
include
:
_normal_right_wrong_template_yaml
task
:
lama_normal_trex_right_wrong
dataset_name
:
TREx
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/_misprimed_true_false_template_yaml
0 → 100755
View file @
97e3c9fe
group: lama_misprimed_true_false
dataset_path: lintang/lama_primed_negated
output_type: multiple_choice
# description: "Answer the following question with either True or False. "
doc_to_text: "Question: {{masked_misprimed[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
doc_to_target: "True"
doc_to_choice: ["True", "False"]
should_decontaminate: true
doc_to_decontamination_query: "Question: {{masked_misprimed[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
metadata:
version: 1.0
lm_eval/tasks/lama_primed_negated/true_false/_negated_true_false_template_yaml
0 → 100755
View file @
97e3c9fe
group: lama_negated_true_false
dataset_path: lintang/lama_primed_negated
output_type: multiple_choice
# description: "Answer the following question with either True or False. "
doc_to_text: "Question: {{masked_negations[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
doc_to_target: "False"
doc_to_choice: ["True", "False"]
should_decontaminate: true
doc_to_decontamination_query: "Question: {{masked_negations[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
metadata:
version: 1.0
lm_eval/tasks/lama_primed_negated/true_false/_normal_true_false_template_yaml
0 → 100755
View file @
97e3c9fe
group: lama_normal_true_false
dataset_path: lintang/lama_primed_negated
output_type: multiple_choice
# description: "Answer the following question with either True or False. "
doc_to_text: "Question: {{masked_sentences[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
doc_to_target: "True"
doc_to_choice: ["True", "False"]
should_decontaminate: true
doc_to_decontamination_query: "Question: {{masked_sentences[0]|replace('[MASK]', obj_label)}} True or False?\n\nAnswer:"
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
metadata:
version: 1.0
lm_eval/tasks/lama_primed_negated/true_false/misprimed_conceptnet.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_true_false_template_yaml
task
:
lama_misprimed_conceptnet_true_false
dataset_name
:
ConceptNet
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/misprimed_google_re.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_true_false_template_yaml
task
:
lama_misprimed_google_re_true_false
dataset_name
:
GoogleRE
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/misprimed_squad.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_true_false_template_yaml
task
:
lama_misprimed_squad_true_false
dataset_name
:
SQUAD
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/misprimed_trex.yaml
0 → 100755
View file @
97e3c9fe
include
:
_misprimed_true_false_template_yaml
task
:
lama_misprimed_trex_true_false
dataset_name
:
TREx
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/negated_conceptnet.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_true_false_template_yaml
task
:
lama_negated_conceptnet_true_false
dataset_name
:
ConceptNet
test_split
:
high_ranked
lm_eval/tasks/lama_primed_negated/true_false/negated_google_re.yaml
0 → 100755
View file @
97e3c9fe
include
:
_negated_true_false_template_yaml
task
:
lama_negated_google_re_true_false
dataset_name
:
GoogleRE
test_split
:
high_ranked
Prev
1
2
3
4
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment