Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
fd9cd80f
Commit
fd9cd80f
authored
May 17, 2024
by
lintangsutawika
Browse files
add test yaml
parent
3be0916c
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
45 additions
and
0 deletions
+45
-0
tests/testyamls/test-01.yaml
tests/testyamls/test-01.yaml
+45
-0
No files found.
tests/testyamls/test-01.yaml
0 → 100644
View file @
fd9cd80f
group
:
test-1
group_alias
:
test
1
task
:
-
piqa
# string task
-
ai2_arc
# string tag
-
task
:
super-glue-lm-eval-v1
# Should this be spread out?
num_fewshot
:
3
-
task
:
swag
# dict registered task
num_fewshot
:
2
-
task
:
mmlu
num_fewshot
:
5
-
group
:
nli-tasks
# dict group
task
:
-
anli
-
boolq
-
sglue_rte
num_fewshot
:
4
metric_list
:
-
metric
:
brier_score
-
task
:
sciq
# dict registered task duplicate
task_alias
:
sciq 2-shot
num_fewshot
:
2
-
task
:
sciq
# dict registered task duplicate
task_alias
:
sciq 4-shot
num_fewshot
:
4
-
task
:
sciq
# dict registered task duplicate
task_alias
:
sciq 6-shot
num_fewshot
:
6
-
task
:
siqa_custom
# dict task
dataset_path
:
social_i_qa
dataset_name
:
null
output_type
:
multiple_choice
training_split
:
train
validation_split
:
validation
doc_to_text
:
"
Question:
{{context}}
{{question}}
\n
Answer:"
target_delimiter
:
"
"
doc_to_choice
:
-
"
{{answerA}}"
-
"
{{answerB}}"
-
"
{{answerC}}"
doc_to_target
:
"
{{
(label|int)
-
1
}}"
metric_list
:
-
metric
:
acc
aggregation
:
mean
higher_is_better
:
true
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment