Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
470e33b1
Commit
470e33b1
authored
Jun 29, 2023
by
Benjamin Fattori
Browse files
add arithmetic tasks
parent
72c61866
Changes
11
Hide whitespace changes
Inline
Side-by-side
Showing
11 changed files
with
150 additions
and
0 deletions
+150
-0
lm_eval/tasks/arithmetic/arithmetic.yaml
lm_eval/tasks/arithmetic/arithmetic.yaml
+0
-0
lm_eval/tasks/arithmetic/arithmetic_1dc.yaml
lm_eval/tasks/arithmetic/arithmetic_1dc.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_2da.yaml
lm_eval/tasks/arithmetic/arithmetic_2da.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_2dm.yaml
lm_eval/tasks/arithmetic/arithmetic_2dm.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_2ds.yaml
lm_eval/tasks/arithmetic/arithmetic_2ds.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_3da.yaml
lm_eval/tasks/arithmetic/arithmetic_3da.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_3ds.yaml
lm_eval/tasks/arithmetic/arithmetic_3ds.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_4da.yaml
lm_eval/tasks/arithmetic/arithmetic_4da.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_4ds.yaml
lm_eval/tasks/arithmetic/arithmetic_4ds.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_5da.yaml
lm_eval/tasks/arithmetic/arithmetic_5da.yaml
+15
-0
lm_eval/tasks/arithmetic/arithmetic_5ds.yaml
lm_eval/tasks/arithmetic/arithmetic_5ds.yaml
+15
-0
No files found.
lm_eval/tasks/arithmetic/arithmetic.yaml
deleted
100644 → 0
View file @
72c61866
lm_eval/tasks/arithmetic/arithmetic_1dc.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_1dc
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_1dc
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_2da.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_2da
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_2da
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_2dm.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_2dm
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_2dm
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_2ds.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_2ds
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_2ds
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_3da.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_3da
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_3da
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_3ds.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_3ds
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_3ds
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_4da.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_4da
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_4da
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_4ds.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_4ds
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_4ds
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_5da.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_5da
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_5da
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
lm_eval/tasks/arithmetic/arithmetic_5ds.yaml
0 → 100644
View file @
470e33b1
group
:
-
arithmetic
task
:
arithmetic_5ds
dataset_path
:
EleutherAI/arithmetic
dataset_name
:
arithmetic_5ds
output_type
:
loglikelihood
validation_split
:
validation
test_split
:
null
template_aliases
:
"
"
doc_to_text
:
"
{{context}}"
doc_to_target
:
"
{{completion}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment