Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
49bfaf68
Commit
49bfaf68
authored
Jul 01, 2025
by
Baber
Browse files
add docs
parent
15d07121
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
14 additions
and
2 deletions
+14
-2
lm_eval/api/task.py
lm_eval/api/task.py
+14
-2
No files found.
lm_eval/api/task.py
View file @
49bfaf68
...
...
@@ -144,7 +144,15 @@ class TemplateConfig:
@
dataclass
class
MCQTemplateConfig
:
"""Encapsulates information about a template."""
"""Encapsulates information about a template.
Would return a sample with the following format:
Question: <doc_to_text(doc)>
A. <doc_to_choice(doc)[0]>
B. <doc_to_choice(doc)[1]>
C. <doc_to_choice(doc)[2]>
D. <doc_to_choice(doc)[3]>
Answer:` doc_to_choice(doc)` for each choice.
"""
doc_to_text
:
Union
[
str
,
Callable
[[
dict
],
str
]]
doc_to_choice
:
Union
[
str
,
list
,
Callable
[[
dict
],
list
]]
...
...
@@ -163,7 +171,11 @@ class MCQTemplateConfig:
@
dataclass
class
ClozeTemplateConfig
:
"""Encapsulates information about a template."""
"""Encapsulates information about a template.
Would return a sample with the following format:
Question: <doc_to_text(doc)>
Answer:` <doc_to_target(doc)>`
"""
doc_to_text
:
Union
[
str
,
Callable
[[
dict
],
str
]]
doc_to_choice
:
Union
[
str
,
list
,
Callable
[[
dict
],
list
]]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment