Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
a9f21c25
Commit
a9f21c25
authored
May 06, 2024
by
haileyschoelkopf
Browse files
add conditional import ; fix task names
parent
e61a3159
Changes
5
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
10 additions
and
11 deletions
+10
-11
lm_eval/tasks/tinyBenchmarks/agg_functions.py
lm_eval/tasks/tinyBenchmarks/agg_functions.py
+9
-1
lm_eval/tasks/tinyBenchmarks/tinyArc.yaml
lm_eval/tasks/tinyBenchmarks/tinyArc.yaml
+0
-2
lm_eval/tasks/tinyBenchmarks/tinyGSM8k.yaml
lm_eval/tasks/tinyBenchmarks/tinyGSM8k.yaml
+0
-3
lm_eval/tasks/tinyBenchmarks/tinyHellaswag.yaml
lm_eval/tasks/tinyBenchmarks/tinyHellaswag.yaml
+0
-2
lm_eval/tasks/tinyBenchmarks/tinyTruthfulQA_mc1.yaml
lm_eval/tasks/tinyBenchmarks/tinyTruthfulQA_mc1.yaml
+1
-3
No files found.
lm_eval/tasks/tinyBenchmarks/agg_functions.py
View file @
a9f21c25
from
typing
import
List
from
typing
import
List
import
numpy
as
np
import
numpy
as
np
import
tinyBenchmarks
as
tb
try
:
import
tinyBenchmarks
as
tb
except
ModuleNotFoundError
:
raise
ModuleNotFoundError
(
"`tinyBenchmarks` is required for tinyBenchmarks task metric calculation, install via
\
`pip install git+https://github.com/felipemaiapolo/tinyBenchmarks`"
)
def
agg_pirt
(
items
:
List
[
float
],
benchmark
:
str
)
->
float
:
def
agg_pirt
(
items
:
List
[
float
],
benchmark
:
str
)
->
float
:
...
...
lm_eval/tasks/tinyBenchmarks/tinyArc.yaml
View file @
a9f21c25
group
:
-
tinyBenchmarks
task
:
tinyArc
task
:
tinyArc
dataset_path
:
tinyBenchmarks/tinyAI2_arc
dataset_path
:
tinyBenchmarks/tinyAI2_arc
dataset_name
:
ARC-Challenge
dataset_name
:
ARC-Challenge
...
...
lm_eval/tasks/tinyBenchmarks/tinyGSM8k.yaml
View file @
a9f21c25
group
:
-
math_word_problems
-
tinyBenchmarks
task
:
tinyGSM8k
task
:
tinyGSM8k
dataset_path
:
tinyBenchmarks/tinyGSM8k
dataset_path
:
tinyBenchmarks/tinyGSM8k
dataset_name
:
main
dataset_name
:
main
...
...
lm_eval/tasks/tinyBenchmarks/tinyHellaswag.yaml
View file @
a9f21c25
group
:
-
tinyBenchmarks
task
:
tinyHellaswag
task
:
tinyHellaswag
dataset_path
:
tinyBenchmarks/tinyHellaswag
dataset_path
:
tinyBenchmarks/tinyHellaswag
dataset_name
:
null
dataset_name
:
null
...
...
lm_eval/tasks/tinyBenchmarks/tinyTruthfulQA_mc1.yaml
View file @
a9f21c25
group
:
task
:
tinyTruthfulQA_mc1
-
truthfulqa
task
:
truthfulqa_mc1
dataset_path
:
tinyBenchmarks/tinyTruthfulQA
dataset_path
:
tinyBenchmarks/tinyTruthfulQA
dataset_name
:
multiple_choice
dataset_name
:
multiple_choice
output_type
:
multiple_choice
output_type
:
multiple_choice
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment