Commit a9f21c25 authored by haileyschoelkopf's avatar haileyschoelkopf
Browse files

add conditional import ; fix task names

parent e61a3159
from typing import List from typing import List
import numpy as np import numpy as np
import tinyBenchmarks as tb
try:
import tinyBenchmarks as tb
except ModuleNotFoundError:
raise ModuleNotFoundError(
"`tinyBenchmarks` is required for tinyBenchmarks task metric calculation, install via \
`pip install git+https://github.com/felipemaiapolo/tinyBenchmarks`"
)
def agg_pirt(items: List[float], benchmark: str) -> float: def agg_pirt(items: List[float], benchmark: str) -> float:
......
group:
- tinyBenchmarks
task: tinyArc task: tinyArc
dataset_path: tinyBenchmarks/tinyAI2_arc dataset_path: tinyBenchmarks/tinyAI2_arc
dataset_name: ARC-Challenge dataset_name: ARC-Challenge
......
group:
- math_word_problems
- tinyBenchmarks
task: tinyGSM8k task: tinyGSM8k
dataset_path: tinyBenchmarks/tinyGSM8k dataset_path: tinyBenchmarks/tinyGSM8k
dataset_name: main dataset_name: main
......
group:
- tinyBenchmarks
task: tinyHellaswag task: tinyHellaswag
dataset_path: tinyBenchmarks/tinyHellaswag dataset_path: tinyBenchmarks/tinyHellaswag
dataset_name: null dataset_name: null
......
group: task: tinyTruthfulQA_mc1
- truthfulqa
task: truthfulqa_mc1
dataset_path: tinyBenchmarks/tinyTruthfulQA dataset_path: tinyBenchmarks/tinyTruthfulQA
dataset_name: multiple_choice dataset_name: multiple_choice
output_type: multiple_choice output_type: multiple_choice
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment