This will set you up with a few `TODO`s to fill-in which we'll now go over in detail.
## Task Heading
Open the file you've just created and add a multiline docstring on the first line with the following contents:
```python
```python
"""
"""
...
@@ -62,102 +71,85 @@ Now let's walk through the actual implementation - from data handling to evaluat
...
@@ -62,102 +71,85 @@ Now let's walk through the actual implementation - from data handling to evaluat
### Downloading your Data
### Downloading your Data
There are 2 standard approaches we follow for downloading data:
All data downloading and management is handled through the HuggingFace (**HF**) [`datasets`](https://github.com/huggingface/datasets) API. So, the first thing you should do is check to see if your task's dataset is already provided in their catalog [here](https://huggingface.co/datasets). If it's not in there, please consider adding it to their Hub to make it accessible to a wider user base by following their [new dataset guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)
.
Now, that you have your HF dataset, you need to assign its path and name to your `Task` in the following fields:
1. Firstly, you should always check to see if your task's dataset is already provided by HuggingFace (__HF__); check their `datasets` catalog [here](https://huggingface.co/datasets). Is it in there? If yes, continue reading here, else go to 2. In the case that it’s there, things are a bit easier. You can inherit from the `HFTask` class as so:
```python
classTaskName(...):
```python
from . common import HFTask
class TaskName(HFTask):
DATASET_PATH="..."
DATASET_PATH="..."
DATASET_NAME="..."
DATASET_NAME="..."
```
```
where `DATASET_PATH` is the name of the benchmark/task dataset as listed by HF and `DATASET_NAME` is the name of, what HF calls, a “data instance” of the benchmark. If your task is not a benchmark containing any data instances just set `DATASET_NAME = None`.
2. Your task's dataset is not in HF's catalog, so you'll have to override a few abstract methods of the `Task` base class. First let's define our benchmark/task and inherit from `Task`.
```python
from lm_eval.base import Task
from pathlib import Path
class TaskName(Task):
DATASET_PATH = Path("data/<task-name>")
```
where `DATASET_PATH` is the local directory we'll download into.
Now we need to override the following methods:
```python
where `DATASET_PATH` is the name of the dataset as listed by HF in the `datasets` Hub and `DATASET_NAME` is the name of, what HF calls, a “data instance” or sub-task of the benchmark. If your task does not contain any data instances, just set `DATASET_NAME = None`.
def download(self):
(If you're familiar with the HF `datasets.load_dataset` function, these are just the first 2 arguments to it.)
```
This should download the dataset into the relative path specified by `DATASET_PATH`. The preferred approach is to use EleutherAI's [best-download](https://github.com/EleutherAI/best-download) package which provides a `download_file` function that lets you validate complete data transmission through a checksum argument. The overall logic should be something like: If the `DATASET_PATH` already exists then don’t download anything and exit the method, otherwise create the `DATASET_PATH` directory and actually download into it. See this [task](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/logiqa.py#L9-L21) for an example.
Next up, we have to set some “flags”:
Next up, we have to set some “flags”:
```python
```python
defhas_training_docs(self):
defhas_training_docs(self):
return# True/False
return# True/False
defhas_validation_docs(self):
defhas_validation_docs(self):
return# True/False
return# True/False
defhas_test_docs(self):
defhas_test_docs(self):
return# True/False
return# True/False
```
```
These methods return `True`/`False` whether or not your task dataset provides documents for each split type. __Note__: if the test set doesn't have publicly available labels, please do not put it down as having a test set.
These methods return `True`/`False` whether or not your task dataset provides documents for each split type. __Note__: if the test set does not have publicly available answer labels, please do not put it down as having a test set - return False.
Lastly, we need to load the documents. In our terminology, a document (`doc`) is a single natural language data example stored in a Python `dict`. E.g.: `{“question”: “What is the capital of France?”, “answer”: “Paris”}`. Override the following methods to load your data splits from their storage location in `DATASET_PATH`:
Lastly, we need to load the documents. In our terminology, a document (`doc`) is a single natural language data example stored in a Python `dict`. E.g.: `{“question”: “What is the capital of France?”, “answer”: “Paris”}`. Override the following methods to load your data splits from their storage location in `DATASET_PATH`:
```python
```python
deftraining_docs(self):
deftraining_docs(self):
return#...
return#...
defvalidation_docs(self):
defvalidation_docs(self):
return#...
return#...
deftest_docs(self):
deftest_docs(self):
return#...
return#...
```
```
These should return a Python iterable (`list` or `generator`) of `dict`s that can be queried for individual `doc` examples. __NOTE__: If your task doesn't have a train/validation/test set, remember to raise a `NotImplementedError` for that specific split.
### Formatting your Few-Shot Examples
These should return a Python iterable (`list` or `generator`) of `dict`s that can be queried for individual `doc` examples.
The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples.
#### Processing Documents
<br>
At this point, you can also process each individual document to, for example, strip whitespace or "detokenize" its fields. Put the processing logic into `_process_doc` and map the functions across training/validation/test docs inside of the respective functions.
🔠 If your task is **multiple-choice**, we require you to format your documents such that they contain `gold` and `choices` fields. They can also have other fields, but those will be ignored by `MultipleChoiceTask`. `choices` should be a list of possible continuations, and `gold` should be an integer specifying the index of the correct completion.
See [this task](https://github.com/EleutherAI/lm-evaluation-harness/blob/6caa0afd96a7a7efb2ec4c1f24ad1756e48f3aa7/lm_eval/tasks/sat.py#L60) for an example. 🔠
⚠️ **Multiple-Choice Formatting**
### Formatting your Few-Shot Examples
If your task is **multiple-choice**, just inherit from the `MultipleChoiceTask` class we provide.
The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples.
```python
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example of type `dict` with `str` key-value members. You should concatenate these `doc` item values together into a neatly formatted prompt.
fromlm_eval.baseimportMultipleChoiceTask
classTaskName(...,MultipleChoiceTask):
```python
defdoc_to_text(self,doc):
return""
```
```
This will require you to format your documents such that they contain `gold` and `choices` fields. They can also have other fields, but those will be ignored by `MultipleChoiceTask`. `choices` should be a list of possible continuations, and `gold` should be an integer specifying the index of the correct completion.
<br>
See [this task](https://github.com/EleutherAI/lm-evaluation-harness/blob/105fa9741ff660f6a62c2eef0d2facfde36dda41/lm_eval/tasks/sat.py#L56) for an example. When used in combination with `HFTask`, it may be useful to override [`_convert_standard`](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/common.py#L28), which will be applied to every document in the HF dataset. See [this task](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/headqa.py) for an example of this.
️🔠 **Multiple-Choice Formatting**
You can now skip ahead to <ahref="#Registering-Your-Task">registering your task</a>.
If your task is multiple-choice, you can now skip ahead to <ahref="#Registering-Your-Task">registering your task</a>.
⚠️ **End Multiple-Choice Formatting**
️️🔠**End Multiple-Choice Formatting**
<br>
<br>
In the case your task is _not_ multiple-choice, override the following methods for your task class:
Format the target answer from the contents of `doc`. Note that the prepended `" "` is required to space out the `doc_to_text` and `doc_to_target` strings.
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example of type `dict` with `str` key-value members. You should concatenate these `doc` item values together into a neatly formatted prompt.
```python
defdoc_to_text(self,doc):
return""
```
Put the target answer of the prompt here, in the form: `" " + <answer>`.
```python
```python
defdoc_to_target(self,doc):
defdoc_to_target(self,doc):
return""
target=""
return" "+target
```
```
Understand that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍.
Finally, be aware that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍.
