Merge branch 'EleutherAI:master' into deepsparselm

README.md

@@ -21,7 +21,7 @@ This project provides a unified framework to test generative language models on
 
 Features:
 
-- 200+ tasks implemented. See the [task-table](./docs/task_table.md) for a complete list.
+- 200+ subtasks / evaluation settings implemented. See the [task-table](./docs/task_table.md) for a complete list.
 - Support for models loaded via [transformers](https://github.com/huggingface/transformers/) (including quantization via [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)), [GPT-NeoX](https://github.com/EleutherAI/gpt-neox), and [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed/), with a flexible tokenization-agnostic interface.
 - Support for commercial APIs including [OpenAI](https://openai.com), [goose.ai](https://goose.ai), and [TextSynth](https://textsynth.com/).
 - Support for evaluation on adapters (e.g. LoRa) supported in [HuggingFace's PEFT library](https://github.com/huggingface/peft).

lm_eval/tasks/bigbench.py

@@ -229,7 +229,11 @@ def create_task_from_path(json_path):
 
 
 def create_all_tasks():
-    resources_dir = importlib.resources.files("lm_eval.datasets") / "bigbench_resources"
+    try:
+        resources_dir = importlib.resources.files("lm_eval.datasets") / "bigbench_resources"
+    except:
+        import importlib_resources
+        resources_dir = importlib_resources.files("lm_eval.datasets") / "bigbench_resources"
     supported_tasks = [os.path.splitext(x)[0] for x in os.listdir(resources_dir)]
     res = {}
     for task_name in supported_tasks:

setup.py

@@ -21,10 +21,11 @@ setuptools.setup(
         "License :: OSI Approved :: MIT License",
         "Operating System :: OS Independent",
     ],
-    python_requires=">=3.9",
+    python_requires=">=3.8",
     install_requires=[
         "datasets>=2.0.0",
         "einops",
+        "importlib-resources",
         "jsonlines",
         "numexpr",
         "openai>=0.6.4",