Merge branch 'main' into big-refactor_dp

README.md

 # Language Model Evaluation Harness
 
+## Announcement
+
+**A new v0.4.0 release of lm-evaluation-harness is available** ! 
+
+New updates and features include:
+
+- Internal refactoring
+- Config-based task creation and configuration
+- Easier import of externally-defined task config files (--include_path, passing path to YAML directly, etc)
+- Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource
+- More advanced configuration options, including output post-processing, answer extraction, and multiple LM generations per document, configurable fewshot settings, and more
+- Speedups and new modeling libraries supported, including: faster data-parallel HF model usage, vLLM support, MPS support with HuggingFace, and more
+- Logging and usability changes
+- New tasks including CoT BIG-Bench-Hard, Belebele, user-defined task groupings, and more
+
+Please see our updated documentation pages in `docs/` for more details.
+
+Development will be continuing on the `main` branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the [EleutherAI discord](discord.gg/eleutherai)!
+
 ## Overview
 
 This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

lm_eval/api/samplers.py

@@ -112,22 +112,3 @@ def get_sampler(name):
         raise ValueError(
             f"Attempted to use contextsampler '{name}', but no sampling strategy for this name found! Supported model names: {', '.join(SAMPLER_REGISTRY.keys())}"
         )
-
-
-# TODO: how should we do design here? might be better to have a single sampler and pass more kwargs at init.
-# Depends what's easier for new user to add own functionality on top of
-
-# types of sampler:
-# - class-balanced, randomly shuffled
-# - class-balanced, one particular set of fewshot examples for all evaled instances
-# - hand-specify number of fewshot examples per class?
-# - random, varies per example (check that this is curr. default in old repo)
-# - random, unified per example
-# - enforce a specific fixed fewshot string! (or should we not use this, in favor of including it in prompt template directly)
-
-
-# - user-specified doc indices to restrict fewshot doc options to
-# - user specifies split to use for drawing fewshot instances (TODO: manually prevent this from being same split you eval!)
-# - user specifies a prepended "description"/string to add in front of the (prompted) input
-
-# - user specifies a location to draw fewshot samples from? DO THIS IN TASK CLASS

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_abstract_algebra.yaml

@@ -35,7 +35,7 @@
   \ then x^2 + c = x^2 + 1 = 0 + 1 for x = 0, 1 + 1 = 2 for x = 1 and 1 + 1 = 2 for\
   \ x = 2, hence x^2 + 1 does not have any roots. For c = 2 the polynomial x^2 + 2\
   \ has two roots at x = 1 and x = 2. Hence Z_3[x]/(x^2 + c) is a field if and only\
-  \ if c = 1. The answer is (B)."
+  \ if c = 1. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_abstract_algebra"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_anatomy.yaml

@@ -51,7 +51,7 @@
   \ of the hyoid bone; therefore, the embryological origin of the hyoid bone are the\
   \ second and the third pharyngeal arches—this information is covered in the last\
   \ option (D). Therefore, we conclude that (D) must be the correct answer. The answer\
-  \ is (D)."
+  \ is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_anatomy"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_astronomy.yaml

@@ -49,7 +49,7 @@
   \ red. Options (C) and (D) are not specific enough about why the color of the surface\
   \ would be red, while (A) is correct because it explains that the surface is red\
   \ due to the rusted materials on the surface and the red color comes from the rust.\
-  \ So the correct option is (A). The answer is (A)."
+  \ So the correct option is (A). The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_astronomy"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_business_ethics.yaml

@@ -50,7 +50,7 @@
   \ that best uses the possible options above is “Beyond the business case for engaging\
   \ the CSR there are a number of moral arguments relating to: negative *externalities*,\
   \ the *power* that corporations possess and the *mutual independence* of business\
-  \ and society. The answer is (D)."
+  \ and society. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_business_ethics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_clinical_knowledge.yaml

@@ -29,7 +29,7 @@
   \ (D) oxidative phosphorylation.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on clinical knowledge for help. The energy for muscular contraction is\
   \ provided by ATP (adenosine triphosphate), which is the powerhouse of the cell.\
-  \ The answer is (A)."
+  \ The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_clinical_knowledge"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_biology.yaml

@@ -55,7 +55,7 @@
   \ resemblance of structures that have different origins, which is not the case for\
   \ the human and bird forearms, which rules out (D). Humans and birds do belong to\
   \ the same clade - a group of organisms composed of a common ancestor. The answer\
-  \ is (C)."
+  \ is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_biology"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_chemistry.yaml

@@ -32,7 +32,7 @@
   \ hyperfine interaction with the 13C (nuclear spin $I = \nrac{1}{2}$) which will\
   \ split the spectrum into 2 lines. This will be further split into 4 lines by the\
   \ interaction with three equivalent 1H nuclei. The total number of lines is therefore\
-  \ $2 \\cdot 4 = 8$. The answer is (E)."
+  \ $2 \\cdot 4 = 8$. The answer is (E).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_chemistry"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_computer_science.yaml

@@ -73,7 +73,7 @@
   Thus we can see that on average a single processor will lock the bus for:\nlock_ns_per_miss\
   \ * misses_per_instruction * instructions_per_ns =\n(1000 nanoseconds / cache miss)\
   \ * (1 cache miss / 50 instructions) * (50 instructions / 27000 nanoseconds) = 1000\
-  \ * (1/50) * (50/27000) = 1000/27000 = 1/27. The answer is (B)."
+  \ * (1/50) * (50/27000) = 1000/27000 = 1/27. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_computer_science"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_mathematics.yaml

@@ -44,7 +44,7 @@
   \ $t \\in \\mathbb{R}, \\ln ((s(t)-2))=-[t / 25]+C$. Let $K:=e^{C}$. Then, for all\
   \ $t \\in \\mathbb{R}$, we have $(s(t))-2=K e^{-t / 25}$, and so $s(t)=2+K e^{-t\
   \ / 25}$. Then $3=s(0)=2+K e^{0}=2+K$, so $K=1$. Then $s(100)=2+K e^{-100 / 25}=2+1\
-  \ \\cdot e^{-4}=2+e^{-4}$. The answer is (D)."
+  \ \\cdot e^{-4}=2+e^{-4}$. The answer is (D).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_mathematics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_medicine.yaml

@@ -46,7 +46,7 @@
   \ monocarbylic acid transporters.\nA: Let's think step by step. We refer to Wikipedia\
   \ articles on medicine for help. Glucose (also known as the blood sugar) is the\
   \ main sugar found in the human body. It is transported into the muscle cell via\
-  \ diffusion through protein transporters called GLUT4. The answer is (A)."
+  \ diffusion through protein transporters called GLUT4. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_other"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_medicine"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml

@@ -38,7 +38,7 @@
   \ go into the gases internal energy or work done against an external force. However,\
   \ if the volume of the gas container is constant, no work will be done (since work\
   \ is pressure times change in volume). So, at constant volume, all of the heat goes\
-  \ into the internal energy. The answer is (B)."
+  \ into the internal energy. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_college_physics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml

@@ -30,7 +30,7 @@
   \ resulted from improper input validation (due to a missing bounds check) in the\
   \ implementation of the TLS heartbeat extension. The vulnerability was classified\
   \ as a buffer over-read, a situation where more data can be read than should be\
-  \ allowed. The answer is (C)."
+  \ allowed. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_computer_security"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml

@@ -27,7 +27,7 @@
   \ speed in the direction of the wind is greater than it would be in the absence\
   \ of wind, and its direction orthogonal to the wind is the same as it would be in\
   \ the absence of the wind. The total speed, which is these two components added\
-  \ in quadrature, is thus greater than the speed in still air. The answer is (B)."
+  \ in quadrature, is thus greater than the speed in still air. The answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_conceptual_physics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_econometrics.yaml

@@ -57,7 +57,7 @@
   \ die away (B) Persist indefinitely (C) Grow exponentially (D) Never occur\nA: Let's\
   \ think step by step. We refer to Wikipedia articles on econometrics for help. This\
   \ is a formal logic problem about stationally process. For a stationary autoregressive\
-  \ process, shocks will eventually die away. The answer is (A)."
+  \ process, shocks will eventually die away. The answer is (A).\n\n"
 "group": "mmlu_flan_cot_fewshot_social_sciences"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_econometrics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_electrical_engineering.yaml

@@ -28,7 +28,7 @@
   \ is 100. Find the total resistance\n(A) 200Ω (B) 100Ω (C) 50Ω (D) 10Ω\nA: Let's\
   \ think step by step. In lap winding, effectively two resistors are connected in\
   \ parallel, so the actual resistance of each pair is 1 Ohm. Since we have 50 pairs,\
-  \ we get a total resistance of 50 Ohms. The answer is (C)."
+  \ we get a total resistance of 50 Ohms. The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_electrical_engineering"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml

@@ -35,7 +35,7 @@
   \nQ: Which expression is equivalent to 5 x 9?\n(A) (5 x 4) x (6 x 5)\n(B) (5 x 5)\
   \ + (5 x 4)\n(C) (5 x 5) + (5 x 9)\n(D) (5 x 9) x (6 x 9)\nA: Let's think step by\
   \ step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5 + 4) = (5 x 5) + (5 x 4). The\
-  \ answer is (B)."
+  \ answer is (B).\n\n"
 "group": "mmlu_flan_cot_fewshot_stem"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_elementary_mathematics"

lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml

@@ -47,7 +47,7 @@
   \ (∀x)(Px ⊃ ~Dx) → For all x, x is on Mars implies that x do not drive on Mars.\n\
   Option (D): ~Dp: → p do not drive on Mars.\nOf all these options, Option (C) appears\
   \ to be the best and most meaningful interpretation of the argument “No people drive\
-  \ on Mars.” The answer is (C)."
+  \ on Mars.” The answer is (C).\n\n"
 "group": "mmlu_flan_cot_fewshot_humanities"
 "include": "_mmlu_flan_cot_fewshot_template_yaml"
 "task": "mmlu_flan_cot_fewshot_formal_logic"