Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
110e65da
Unverified
Commit
110e65da
authored
Mar 20, 2025
by
Baber Abbasi
Committed by
GitHub
Mar 20, 2025
Browse files
fix typo (#2820)
parent
61b63da7
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
7 additions
and
2 deletions
+7
-2
lm_eval/tasks/humaneval/README.md
lm_eval/tasks/humaneval/README.md
+1
-0
lm_eval/tasks/humaneval/humaneval_64_instruct.yaml
lm_eval/tasks/humaneval/humaneval_64_instruct.yaml
+3
-1
lm_eval/tasks/humaneval/humaneval_instruct.yaml
lm_eval/tasks/humaneval/humaneval_instruct.yaml
+3
-1
No files found.
lm_eval/tasks/humaneval/README.md
View file @
110e65da
...
@@ -49,3 +49,4 @@ If other tasks on this dataset are already supported:
...
@@ -49,3 +49,4 @@ If other tasks on this dataset are already supported:
*
[ ] Have you noted which, if any, published evaluation setups are matched by this variant?
*
[ ] Have you noted which, if any, published evaluation setups are matched by this variant?
### Changelog
### Changelog
v2 20-MAR-2025:
`humaneval_instruct`
,
`humaneval_instruct_64`
: fixed typo in gen_prefix
lm_eval/tasks/humaneval/humaneval_64_instruct.yaml
View file @
110e65da
include
:
humaneval_64.yaml
include
:
humaneval_64.yaml
task
:
humaneval_64_instruct
task
:
humaneval_64_instruct
doc_to_text
:
"
Write
a
solution
to
the
following
problem
and
make
sure
that
it
passes
the
tests:
\n
```{{prompt}}"
doc_to_text
:
"
Write
a
solution
to
the
following
problem
and
make
sure
that
it
passes
the
tests:
\n
```{{prompt}}"
gen_prefix
:
"
Here
is
the
completed
function:
\
\
n```python
\
\
n{{prompt}}
\
\
n"
gen_prefix
:
"
Here
is
the
completed
function:
\n
```python
\n
{{prompt}}
\n
"
filter_list
:
filter_list
:
-
name
:
"
create_test"
-
name
:
"
create_test"
filter
:
filter
:
-
function
:
"
custom"
-
function
:
"
custom"
filter_fn
:
!function
utils.build_predictions_instruct
filter_fn
:
!function
utils.build_predictions_instruct
metadata
:
version
:
2.0
lm_eval/tasks/humaneval/humaneval_instruct.yaml
View file @
110e65da
include
:
humaneval.yaml
include
:
humaneval.yaml
task
:
humaneval_instruct
task
:
humaneval_instruct
doc_to_text
:
"
Write
a
solution
to
the
following
problem
and
make
sure
that
it
passes
the
tests:
\n
```{{prompt}}"
doc_to_text
:
"
Write
a
solution
to
the
following
problem
and
make
sure
that
it
passes
the
tests:
\n
```{{prompt}}"
gen_prefix
:
"
Here
is
the
completed
function:
\
\
n```python
\
\
n{{prompt}}
\
\
n"
gen_prefix
:
"
Here
is
the
completed
function:
\n
```python
\n
{{prompt}}
\n
"
filter_list
:
filter_list
:
-
name
:
"
create_test"
-
name
:
"
create_test"
filter
:
filter
:
-
function
:
"
custom"
-
function
:
"
custom"
filter_fn
:
!function
utils.build_predictions_instruct
filter_fn
:
!function
utils.build_predictions_instruct
metadata
:
version
:
2.0
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment