@@ -112,7 +112,9 @@ after this go <a href="#Registering-Your-Task">register your task</a>.
...
@@ -112,7 +112,9 @@ after this go <a href="#Registering-Your-Task">register your task</a>.
### Versioning
### Versioning
Tasks in the harness can always evolve. Metrics get updated, data sources change, etc. It’s important to mark each task with a version attribute so users can specify which version of the implementation was used to obtain their results. Add a `VERSION` attribute to your task class set to 0 (the first version of your task):
Tasks in the harness can always evolve. Metrics get updated, data sources
change, etc. It’s important to mark each task with a version attribute so users
can document which implementation version was used to obtain their results. Add a `VERSION` attribute to your task class set to 0 (the first version of your task):
```python
```python
classTaskName(...):
classTaskName(...):
...
@@ -122,23 +124,27 @@ class TaskName(...):
...
@@ -122,23 +124,27 @@ class TaskName(...):
### Formatting your Few-Shot Examples
### Formatting your Few-Shot Examples
The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples. Override the following methods for your task class:
The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples. Override the following methods for your task class:
Put the natural language task description as a single line (no `\n`s) string here. E.g. `"Translate English to French:"`
```python
```python
deffewshot_description(self):
deffewshot_description(self):
return""
return""
```
```
Put your natural language task description as a single line (no `\n`s) string here. E.g. `"Translate English to French:"`
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example (in dictionary form) . You should concatenate its members into a nicely formatted prompt.
```python
```python
defdoc_to_text(self,doc):
defdoc_to_text(self,doc):
return""
return""
```
```
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example (in dictionary form) . You should concatenate its members into a nicely formatted prompt.
Put the target answer of the prompt here, in the form: `" " + <answer>`.
```python
```python
defdoc_to_target(self,doc):
defdoc_to_target(self,doc):
return""
return""
```
```
Put the target answer of the prompt here, in the form: `" " + <answer>`.
Understand that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍.
Understand that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍.
...
@@ -159,7 +165,8 @@ python -m scripts.write_out \
...
@@ -159,7 +165,8 @@ python -m scripts.write_out \
--num_examples N
--num_examples N
```
```
Open the file specified at the `--output_base_path <path>` and ensure it passes the eye test.
Open the file specified at the `--output_base_path <path>` and ensure it passes