Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
bbb8386c
Commit
bbb8386c
authored
Apr 16, 2024
by
lintangsutawika
Browse files
removed alt worlds prompts
parent
3e5e9da2
Changes
1000
Hide whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
0 additions
and
204 deletions
+0
-204
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_02/c.yaml
...ernative_worlds/arc_easy/output_variation/style_02/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/a.yaml
...ernative_worlds/arc_easy/output_variation/style_03/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/b.yaml
...ernative_worlds/arc_easy/output_variation/style_03/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/c.yaml
...ernative_worlds/arc_easy/output_variation/style_03/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/a.yaml
...ernative_worlds/arc_easy/output_variation/style_04/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/b.yaml
...ernative_worlds/arc_easy/output_variation/style_04/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/c.yaml
...ernative_worlds/arc_easy/output_variation/style_04/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/a.yaml
...ernative_worlds/arc_easy/output_variation/style_05/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/b.yaml
...ernative_worlds/arc_easy/output_variation/style_05/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/c.yaml
...ernative_worlds/arc_easy/output_variation/style_05/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/a.yaml
...ernative_worlds/arc_easy/output_variation/style_06/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/b.yaml
...ernative_worlds/arc_easy/output_variation/style_06/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/c.yaml
...ernative_worlds/arc_easy/output_variation/style_06/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/a.yaml
...ernative_worlds/arc_easy/output_variation/style_07/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/b.yaml
...ernative_worlds/arc_easy/output_variation/style_07/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/c.yaml
...ernative_worlds/arc_easy/output_variation/style_07/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/a.yaml
...ernative_worlds/arc_easy/output_variation/style_08/a.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/b.yaml
...ernative_worlds/arc_easy/output_variation/style_08/b.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/c.yaml
...ernative_worlds/arc_easy/output_variation/style_08/c.yaml
+0
-6
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/styles.py
...rc/alternative_worlds/arc_easy/output_variation/styles.py
+0
-90
No files found.
Too many changes to show.
To preserve performance only
1000 of 1000+
files are displayed.
Plain diff
Email patch
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_02/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_02
task
:
arc_easy_alt_ov_02c
doc_to_text
:
!function
../styles.template_02
doc_to_choice
:
!function
../styles.choice_02c
doc_to_decontamination_query
:
!function
../styles.template_02
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_03
task
:
arc_easy_alt_ov_03a
doc_to_text
:
!function
../styles.template_03
doc_to_choice
:
!function
../styles.choice_03a
doc_to_decontamination_query
:
!function
../styles.template_03
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_03
task
:
arc_easy_alt_ov_03b
doc_to_text
:
!function
../styles.template_03
doc_to_choice
:
!function
../styles.choice_03b
doc_to_decontamination_query
:
!function
../styles.template_03
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_03/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_03
task
:
arc_easy_alt_ov_03c
doc_to_text
:
!function
../styles.template_03
doc_to_choice
:
!function
../styles.choice_03c
doc_to_decontamination_query
:
!function
../styles.template_03
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_04
task
:
arc_easy_alt_ov_04a
doc_to_text
:
!function
../styles.template_04
doc_to_choice
:
!function
../styles.choice_04a
doc_to_decontamination_query
:
!function
../styles.template_04
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_04
task
:
arc_easy_alt_ov_04b
doc_to_text
:
!function
../styles.template_04
doc_to_choice
:
!function
../styles.choice_04b
doc_to_decontamination_query
:
!function
../styles.template_04
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_04/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_04
task
:
arc_easy_alt_ov_04c
doc_to_text
:
!function
../styles.template_04
doc_to_choice
:
!function
../styles.choice_04c
doc_to_decontamination_query
:
!function
../styles.template_04
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_05
task
:
arc_easy_alt_ov_05a
doc_to_text
:
!function
../styles.template_05
doc_to_choice
:
!function
../styles.choice_05a
doc_to_decontamination_query
:
!function
../styles.template_05
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_05
task
:
arc_easy_alt_ov_05b
doc_to_text
:
!function
../styles.template_05
doc_to_choice
:
!function
../styles.choice_05b
doc_to_decontamination_query
:
!function
../styles.template_05
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_05/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_05
task
:
arc_easy_alt_ov_05c
doc_to_text
:
!function
../styles.template_05
doc_to_choice
:
!function
../styles.choice_05c
doc_to_decontamination_query
:
!function
../styles.template_05
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_06
task
:
arc_easy_alt_ov_06a
doc_to_text
:
!function
../styles.template_06
doc_to_choice
:
!function
../styles.choice_06a
doc_to_decontamination_query
:
!function
../styles.template_06
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_06
task
:
arc_easy_alt_ov_06b
doc_to_text
:
!function
../styles.template_06
doc_to_choice
:
!function
../styles.choice_06b
doc_to_decontamination_query
:
!function
../styles.template_06
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_06/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_06
task
:
arc_easy_alt_ov_06c
doc_to_text
:
!function
../styles.template_06
doc_to_choice
:
!function
../styles.choice_06c
doc_to_decontamination_query
:
!function
../styles.template_06
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_07
task
:
arc_easy_alt_ov_07a
doc_to_text
:
!function
../styles.template_07
doc_to_choice
:
!function
../styles.choice_07a
doc_to_decontamination_query
:
!function
../styles.template_07
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_07
task
:
arc_easy_alt_ov_07b
doc_to_text
:
!function
../styles.template_07
doc_to_choice
:
!function
../styles.choice_07b
doc_to_decontamination_query
:
!function
../styles.template_07
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_07/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_07
task
:
arc_easy_alt_ov_07c
doc_to_text
:
!function
../styles.template_07
doc_to_choice
:
!function
../styles.choice_07c
doc_to_decontamination_query
:
!function
../styles.template_07
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/a.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_08
task
:
arc_easy_alt_ov_08a
doc_to_text
:
!function
../styles.template_08
doc_to_choice
:
!function
../styles.choice_08a
doc_to_decontamination_query
:
!function
../styles.template_08
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/b.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_08
task
:
arc_easy_alt_ov_08b
doc_to_text
:
!function
../styles.template_08
doc_to_choice
:
!function
../styles.choice_08b
doc_to_decontamination_query
:
!function
../styles.template_08
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/style_08/c.yaml
deleted
100644 → 0
View file @
3e5e9da2
include
:
../_arc_easy_alt_yaml
group
:
arc_easy_alt_ov_08
task
:
arc_easy_alt_ov_08c
doc_to_text
:
!function
../styles.template_08
doc_to_choice
:
!function
../styles.choice_08c
doc_to_decontamination_query
:
!function
../styles.template_08
lm_eval/tasks/arc/alternative_worlds/arc_easy/output_variation/styles.py
deleted
100644 → 0
View file @
3e5e9da2
import
string
from
functools
import
partial
def
doc_to_text_base
(
alphabet
,
style
,
doc
):
choices
=
doc
[
"choices"
][
"text"
]
num
=
len
(
choices
)
letter_list
=
[
style
.
format
(
letter
)
for
letter
in
alphabet
[
0
:
num
]]
if
"
\t
"
in
style
:
choice_string
=
"{}{}"
else
:
choice_string
=
"{} {}"
doc_to_text
=
"
\n
"
.
join
(
[
"Question: "
+
doc
[
"question"
],
]
+
[
choice_string
.
format
(
i
,
j
)
for
i
,
j
in
zip
(
letter_list
,
choices
)]
+
[
"Answer:"
]
)
return
doc_to_text
# Full continuation
def
choice_A
(
doc
):
return
doc
[
"choices"
][
"text"
]
# Letters only
def
choice_B
(
alphabet
,
style
,
doc
):
choices
=
doc
[
"choices"
][
"text"
]
num
=
len
(
choices
)
letter_list
=
[
style
.
format
(
letter
)
for
letter
in
alphabet
[
0
:
num
]]
if
"
\t
"
in
style
:
letter_list
=
[
letter
.
replace
(
"
\t
"
,
""
)
for
letter
in
letter_list
]
return
letter_list
# Letters + Full continuation
def
choice_C
(
alphabet
,
style
,
doc
):
choices
=
doc
[
"choices"
][
"text"
]
num
=
len
(
choices
)
letter_list
=
[
style
.
format
(
letter
)
for
letter
in
alphabet
[
0
:
num
]]
if
"
\t
"
not
in
style
:
letter_list
=
[
letter
+
" "
for
letter
in
letter_list
]
return
[
letter
+
choice
for
letter
,
choice
in
zip
(
letter_list
,
choices
)]
template_01
=
partial
(
doc_to_text_base
,
string
.
ascii_lowercase
,
"({})"
)
choice_01a
=
choice_A
choice_01b
=
partial
(
choice_B
,
string
.
ascii_lowercase
,
"({})"
)
choice_01c
=
partial
(
choice_C
,
string
.
ascii_lowercase
,
"({})"
)
template_02
=
partial
(
doc_to_text_base
,
string
.
ascii_lowercase
,
"{})"
)
choice_02a
=
choice_A
choice_02b
=
partial
(
choice_B
,
string
.
ascii_lowercase
,
"{})"
)
choice_02c
=
partial
(
choice_C
,
string
.
ascii_lowercase
,
"{})"
)
template_03
=
partial
(
doc_to_text_base
,
string
.
ascii_lowercase
,
"{}."
)
choice_03a
=
choice_A
choice_03b
=
partial
(
choice_B
,
string
.
ascii_lowercase
,
"{}."
)
choice_03c
=
partial
(
choice_C
,
string
.
ascii_lowercase
,
"{}."
)
template_04
=
partial
(
doc_to_text_base
,
string
.
ascii_lowercase
,
"{}
\t
"
)
choice_04a
=
choice_A
choice_04b
=
partial
(
choice_B
,
string
.
ascii_lowercase
,
"{}
\t
"
)
choice_04c
=
partial
(
choice_C
,
string
.
ascii_lowercase
,
"{}
\t
"
)
template_05
=
partial
(
doc_to_text_base
,
string
.
ascii_uppercase
,
"({})"
)
choice_05a
=
choice_A
choice_05b
=
partial
(
choice_B
,
string
.
ascii_uppercase
,
"({})"
)
choice_05c
=
partial
(
choice_C
,
string
.
ascii_uppercase
,
"({})"
)
template_06
=
partial
(
doc_to_text_base
,
string
.
ascii_uppercase
,
"{})"
)
choice_06a
=
choice_A
choice_06b
=
partial
(
choice_B
,
string
.
ascii_uppercase
,
"{})"
)
choice_06c
=
partial
(
choice_C
,
string
.
ascii_uppercase
,
"{})"
)
template_07
=
partial
(
doc_to_text_base
,
string
.
ascii_uppercase
,
"{}."
)
choice_07a
=
choice_A
choice_07b
=
partial
(
choice_B
,
string
.
ascii_uppercase
,
"{}."
)
choice_07c
=
partial
(
choice_C
,
string
.
ascii_uppercase
,
"{}."
)
template_08
=
partial
(
doc_to_text_base
,
string
.
ascii_uppercase
,
"{}
\t
"
)
choice_08a
=
choice_A
choice_08b
=
partial
(
choice_B
,
string
.
ascii_uppercase
,
"{}
\t
"
)
choice_08c
=
partial
(
choice_C
,
string
.
ascii_uppercase
,
"{}
\t
"
)
Prev
1
2
3
4
5
6
7
…
50
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment