Merged with master

README.md

@@ -21,7 +21,7 @@ This project provides a unified framework to test generative language models on
 
 Features:
 
-- 200+ tasks implemented. See the [task-table](./docs/task_table.md) for a complete list.
+- 200+ subtasks / evaluation settings implemented. See the [task-table](./docs/task_table.md) for a complete list.
 - Support for models loaded via [transformers](https://github.com/huggingface/transformers/) (including quantization via [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)), [GPT-NeoX](https://github.com/EleutherAI/gpt-neox), and [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed/), with a flexible tokenization-agnostic interface.
 - Support for commercial APIs including [OpenAI](https://openai.com), [goose.ai](https://goose.ai), and [TextSynth](https://textsynth.com/).
 - Support for evaluation on adapters (e.g. LoRa) supported in [HuggingFace's PEFT library](https://github.com/huggingface/peft).

lm_eval/tasks/bigbench.py

@@ -233,6 +233,7 @@ def create_all_tasks():
     try:
         resources_dir = importlib.resources.files("lm_eval.datasets") / "bigbench_resources"
     except:
+        import importlib_resources
         resources_dir = importlib_resources.files("lm_eval.datasets") / "bigbench_resources"
     supported_tasks = [os.path.splitext(x)[0] for x in os.listdir(resources_dir)]
     res = {}

setup.py

@@ -25,6 +25,7 @@ setuptools.setup(
     install_requires=[
         "datasets>=2.0.0",
         "einops",
+        "importlib-resources",
         "jsonlines",
         "numexpr",
         "openai>=0.6.4",