Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
fe9c5707
Unverified
Commit
fe9c5707
authored
Jan 29, 2025
by
Baber Abbasi
Committed by
GitHub
Jan 29, 2025
Browse files
remove `group` from bigbench task configs (#2663)
* remove group from task configs * add tags * update readme
parent
4b4b0363
Changes
4
Show whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
9 additions
and
3 deletions
+9
-3
lm_eval/tasks/bigbench/README.md
lm_eval/tasks/bigbench/README.md
+6
-0
lm_eval/tasks/bigbench/generate_until_template_yaml
lm_eval/tasks/bigbench/generate_until_template_yaml
+1
-1
lm_eval/tasks/bigbench/multiple_choice_template_a_yaml
lm_eval/tasks/bigbench/multiple_choice_template_a_yaml
+1
-1
lm_eval/tasks/bigbench/multiple_choice_template_b_yaml
lm_eval/tasks/bigbench/multiple_choice_template_b_yaml
+1
-1
No files found.
lm_eval/tasks/bigbench/README.md
View file @
fe9c5707
...
@@ -30,6 +30,12 @@ Homepage: https://github.com/google/BIG-bench
...
@@ -30,6 +30,12 @@ Homepage: https://github.com/google/BIG-bench
*
`group_name`
:
`Short description`
*
`group_name`
:
`Short description`
#### Tags
*
`bigbench_generate_until`
*
`bigbench_multiple_choice_a`
*
`bigbench_multiple_choice_b`
#### Tasks
#### Tasks
*
`task_name`
:
`1-sentence description of what this particular task does`
*
`task_name`
:
`1-sentence description of what this particular task does`
...
...
lm_eval/tasks/bigbench/generate_until_template_yaml
View file @
fe9c5707
group
: bigbench_generate_until
tag
: bigbench_generate_until
dataset_path: hails/bigbench
dataset_path: hails/bigbench
output_type: generate_until
output_type: generate_until
dataset_kwargs:
dataset_kwargs:
...
...
lm_eval/tasks/bigbench/multiple_choice_template_a_yaml
View file @
fe9c5707
group
: bigbench_multiple_choice
tag
: bigbench_multiple_choice
_a
dataset_path: hails/bigbench
dataset_path: hails/bigbench
dataset_kwargs:
dataset_kwargs:
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
...
...
lm_eval/tasks/bigbench/multiple_choice_template_b_yaml
View file @
fe9c5707
group
: bigbench_multiple_choice
tag
: bigbench_multiple_choice
_b
dataset_path: hails/bigbench
dataset_path: hails/bigbench
dataset_kwargs:
dataset_kwargs:
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
# num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment