"With the vast amount of work done in the field today, it helps to have a tool that people can use easily to share their results and use to check others to ensure reported numbers are valid. The LM Evaluation Harness is one such tool the community has used extensively. We want to continue to support the community and with that in mind, we’re excited to announce a major update on the LM Evaluation Harness to further our goal for open and accessible AI research."
],
"metadata": {
"id": "Z7k2vq1iAdqr"
}
},
{
"cell_type": "markdown",
"source": [
"Our refactor stems from our beliefs of the following that we think are best practices\n",
"\n",
"1. Never Copy Results from Other Papers\n",
"2. Always share your exact prompts\n",
"3. Always provide model outputs\n",
"4. Qualitatively review a small batch of outputs before running evaluation jobs at scale\n"
],
"metadata": {
"id": "0gDoM0AJAvEc"
}
},
{
"cell_type": "markdown",
"source": [
"In this notebook we will be going through a short tutorial on how things work."
"Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (3.3.2)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (6.0.4)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (4.0.3)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.9.2)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.4.0)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.3.1)\n",
"Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm-eval==1.0.0) (5.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (2023.7.22)\n",
"Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval==1.0.0) (2.8.2)\n",
"Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval==1.0.0) (2023.3.post1)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.8->lm-eval==1.0.0) (2.1.3)\n",
"Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score>=0.0.4->lm-eval==1.0.0) (8.1.7)\n",
"Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.8->lm-eval==1.0.0) (1.3.0)\n",
"Building wheels for collected packages: lm-eval, rouge-score, sqlitedict\n",
" Building wheel for lm-eval (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for lm-eval: filename=lm_eval-1.0.0-py3-none-any.whl size=986403 sha256=b7907b5840136fbd6ae948150ebab82776a0deb76359e59da24fbd2259de8094\n",
" Stored in directory: /tmp/pip-ephem-wheel-cache-axf286nt/wheels/17/01/26/599c0779e9858a70a73fa8a306699b5b9a868f820c225457b0\n",
" Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24933 sha256=8983ef5e600070817b0114ebe7ad48d72891d918ab2beea00ed6c5c437a6f713\n",
" Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
" Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=146f7dcb0d03ca6c21a08e9d93d74d6c07cd42d0e37ddb7e922127e5ddb5e213\n",
" Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
"Successfully built lm-eval rouge-score sqlitedict\n",
"## Make task evaluations with configurable tasks\n",
"\n",
"Even within the same task, many works have reported numbers based on different choices of evaluation. Some report on the test sets, validation sets, or even subset of the training sets. Others have specialized prompts and verbalizers. We introduce YAMLs to allow users to easily make different variations. By leveraging the YAML configs to configure evaluations, the refactored LM-Eval takes the methods of the `Tasks` object and makes them configurable by setting the appropriate attributes in the config file. There, users can set the tasks they want by setting the name of the HF dataset (local tasks are also possible) the dataset splits used and much more. Key configurations such as `doc_to_text`, previously implemented as a method of the same name, is now configurable with jinja2 to allow high-level scripting to transform a HF dataset to text string as input to the model.\n",
"\n"
],
"metadata": {
"id": "8rfUeX6n_wkK"
}
},
{
"cell_type": "markdown",
"source": [
"A core-feature to LM-Eval is to configure tasks with YAML configs. With configs, fill preset fields to easily setup a task."
"2023-11-27:08:14:53,517 INFO [utils.py:160] NumExpr defaulting to 2 threads.\n",
"2023-11-27 08:14:54.499605: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-11-27 08:14:54.499658: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-11-27 08:14:54.499691: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
"2023-11-27 08:14:56.139266: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"Oftenly, tasks are part of a larger group used to meausre different capabilities. The dynamism of the field today means new dimensions of evaluation can come about which would mix and match new and older tasks alike. In LM-Eval, We can also group tasks and call that the group name to evaluate on a set of tasks easily. In this instance, let's evaluate the group `yes_or_no_tasks` which comprise of the tasks `demo_boolq` and `demo_cola`; tasks which are multiple choice tasks with options `yes` and `no` as the name suggests.\n",
"\n",
"<!-- making new groups is easier than ever, allowing user to work bottom-up by makiing individual tasks and linking them to a group or Top-Down, making a new group by listing existing tasks.\n",
"\n",
"We also show the aggregate across samples besides only showing the aggregation between subtasks. This may come in handy when certain groups want to be aggregated as a single task. -->\n",
"\n",
"\n"
],
"metadata": {
"id": "LOUHK7PtQfq4"
}
},
{
"cell_type": "code",
"source": [
"YAML_cola_string = '''\n",
"group: yes_or_no_tasks\n",
"task: demo_cola\n",
"dataset_path: glue\n",
"dataset_name: cola\n",
"output_type: multiple_choice\n",
"training_split: train\n",
"validation_split: validation\n",
"doc_to_text: \"{{sentence}}\\nQuestion: Does this sentence make sense?\\nAnswer:\"\n",
"2023-11-27:08:15:04,956 INFO [utils.py:160] NumExpr defaulting to 2 threads.\n",
"2023-11-27 08:15:06.059715: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-11-27 08:15:06.059774: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-11-27 08:15:06.059810: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
"2023-11-27 08:15:07.648550: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"2023-11-27:08:15:10,952 INFO [__main__.py:124] Verbosity set to INFO\n",
"2023-11-27:08:15:18,895 WARNING [__main__.py:130] --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
"2023-11-27:08:15:18,895 INFO [__main__.py:135] Including path: ./\n",
"2023-11-27:08:15:18,915 INFO [__main__.py:197] Selected Tasks: ['yes_or_no_tasks']\n",
"2023-11-27:08:15:18,979 INFO [huggingface.py:119] Using device 'cuda'\n",
"2023-11-27:08:16:31,344 WARNING [task.py:612] [Task: demo_boolq] metric acc is defined, but aggregation is not. using default aggregation=mean\n",
"2023-11-27:08:16:31,344 WARNING [task.py:624] [Task: demo_boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True\n",
"Generating test split: 100% 3245/3245 [00:00<00:00, 20505.42 examples/s]\n",
"2023-11-27:08:16:35,225 WARNING [task.py:612] [Task: demo_cola] metric acc is defined, but aggregation is not. using default aggregation=mean\n",
"2023-11-27:08:16:35,225 WARNING [task.py:624] [Task: demo_cola] metric acc is defined, but higher_is_better is not. using default higher_is_better=True\n",
"The following is a yaml made to evaluate the specific subtask of `high_school_geography` from MMLU. It uses the standard prompt where the we choose the letters from the options with most likelihood as the model's prediction."
],
"metadata": {
"id": "XceRKCuuDtbn"
}
},
{
"cell_type": "code",
"source": [
"YAML_mmlu_geo_string = '''\n",
"group: mmlu\n",
"task: demo_mmlu_high_school_geography\n",
"dataset_path: cais/mmlu\n",
"dataset_name: high_school_geography\n",
"description: \"The following are multiple choice questions (with answers) about high school geography.\\n\\n\"\n",
"2023-11-27:08:16:50,338 INFO [utils.py:160] NumExpr defaulting to 2 threads.\n",
"2023-11-27 08:16:51.182014: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-11-27 08:16:51.182064: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-11-27 08:16:51.182100: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
"2023-11-27 08:16:52.470541: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"2023-11-27:08:16:55,648 INFO [__main__.py:124] Verbosity set to INFO\n",
"2023-11-27:08:17:05,419 WARNING [__main__.py:130] --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
"2023-11-27:08:17:05,420 INFO [__main__.py:135] Including path: ./\n",
"2023-11-27:08:17:05,433 INFO [__main__.py:197] Selected Tasks: ['demo_mmlu_high_school_geography']\n",
"2023-11-27:08:17:05,469 INFO [huggingface.py:119] Using device 'cuda'\n",
"We could also evaluate this task in a different way. For example, instead of observing the loglikelihood of the letters, we can instead evaluate on the choices themselves as the continuation. This is done by simply changing `doc_to_choice` from a list of letters to the corresponding `choices` field from the HF dataset. We write `\"{{choices}}\"` so that the string field is interpreted as jinja string that acquires the list from the HF dataset directly.\n",
"\n",
"Another convinient feature here is since we're only modifying the `doc_to_choice` and the rest of config is the same as the task above, we can use the above configuration as a template by using `include: mmlu_high_school_geography.yaml` to load the config from that file. We'll need to add a unique task name as to not colide with the existing yaml config we're including. For this case we'll simply name this one `mmlu_high_school_geography_continuation`. `doc_to_text` is added here just for sake of clarity."
"2023-11-27:08:17:52,429 INFO [utils.py:160] NumExpr defaulting to 2 threads.\n",
"2023-11-27 08:17:53.714003: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-11-27 08:17:53.714080: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-11-27 08:17:53.714119: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
"2023-11-27 08:17:55.953811: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"2023-11-27:08:17:59,577 INFO [__main__.py:124] Verbosity set to INFO\n",
"2023-11-27:08:18:07,654 WARNING [__main__.py:130] --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
"2023-11-27:08:18:07,654 INFO [__main__.py:135] Including path: ./\n",
"2023-11-27:08:18:07,686 INFO [__main__.py:197] Selected Tasks: ['demo_mmlu_high_school_geography_continuation']\n",
"2023-11-27:08:18:07,735 INFO [huggingface.py:119] Using device 'cuda'\n",
"2023-11-27:08:18:25,609 INFO [task.py:353] Building contexts for task on rank 0...\n",
"2023-11-27:08:18:25,628 INFO [evaluator.py:290] Running loglikelihood requests\n",
"100% 40/40 [00:05<00:00, 7.68it/s]\n",
"fatal: not a git repository (or any of the parent directories): .git\n",
"To prepare a task we can simply fill in a YAML config with the relevant information.\n",
"\n",
"`output_type`\n",
"The current provided evaluation types comprise of the following:\n",
"1. `loglikelihood`: Evaluates the loglikeihood of a continuation\n",
"2. `loglikelihood_rolling`\n",
"3. `multiple_choice`: Evaluates loglikelihood among the a number of choices predicted by the model.\n",
"4. `greedy_until`: Model outputs greedy generation (can be configured to to use beam search and other generation-related paramaters)\n",
"\n",
"The core prompt revolves around 3 fields.\n",
"1. `doc_to_text`: Denotes the prompt template that will be used as input to the model.\n",
"2. `doc_to_choice`: Available choices that will be used as continuation for the model. This is used when the `output_type` is `multiple_choice`.\n",
"3. `doc_to_target`: When `output_type` is `multiple_choice` this can be an index that corresponds to the correct answer or the answer string itself (must be a subset of `doc_to_choice`). For other tasks, this is expected to be a string. You can fill this field with a feature name from the HF dataset so long as the resulting feature follows the conditioned described.\n",
"\n",
"<!-- Advanced notes:\n",
"In some cases, like Winograd, we want to alternate the left-hand side of a prompt and keep the answer the same -->\n"
],
"metadata": {
"id": "duBDqC6PAdjL"
}
},
{
"cell_type": "markdown",
"source": [
"## What if Jinja is not Sufficient?\n",
"\n",
"There can be times where Jinja is not enough to make the prompt we had in mind.\n",
"\n",
"1. Use `!function` operator for the prompt-related fields to pass a python function to build the prompt template.\n",
"2023-11-27:08:18:37,991 INFO [utils.py:160] NumExpr defaulting to 2 threads.\n",
"2023-11-27 08:18:39.666844: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-11-27 08:18:39.666894: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-11-27 08:18:39.666949: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
"2023-11-27 08:18:41.101009: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"2023-11-27:08:18:44,452 INFO [__main__.py:124] Verbosity set to INFO\n",
"2023-11-27:08:18:54,278 WARNING [__main__.py:130] --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
"2023-11-27:08:18:54,279 INFO [__main__.py:135] Including path: ./\n",
"2023-11-27:08:18:54,303 INFO [__main__.py:197] Selected Tasks: ['demo_mmlu_high_school_geography_function_prompt']\n",
"2023-11-27:08:18:54,333 INFO [huggingface.py:119] Using device 'cuda'\n",
"2023-11-27:08:19:12,078 INFO [task.py:353] Building contexts for task on rank 0...\n",
"2023-11-27:08:19:12,084 INFO [evaluator.py:290] Running loglikelihood requests\n",
"100% 40/40 [00:02<00:00, 16.47it/s]\n",
"fatal: not a git repository (or any of the parent directories): .git\n",
