Commit 011cc891 authored by Jonathan Tow's avatar Jonathan Tow
Browse files

fix up some wording

parent 00cecd59
...@@ -112,7 +112,9 @@ after this go <a href="#Registering-Your-Task">register your task</a>. ...@@ -112,7 +112,9 @@ after this go <a href="#Registering-Your-Task">register your task</a>.
### Versioning ### Versioning
Tasks in the harness can always evolve. Metrics get updated, data sources change, etc. It’s important to mark each task with a version attribute so users can specify which version of the implementation was used to obtain their results. Add a `VERSION` attribute to your task class set to 0 (the first version of your task): Tasks in the harness can always evolve. Metrics get updated, data sources
change, etc. It’s important to mark each task with a version attribute so users
can document which implementation version was used to obtain their results. Add a `VERSION` attribute to your task class set to 0 (the first version of your task):
```python ```python
class TaskName(...): class TaskName(...):
...@@ -122,23 +124,27 @@ class TaskName(...): ...@@ -122,23 +124,27 @@ class TaskName(...):
### Formatting your Few-Shot Examples ### Formatting your Few-Shot Examples
The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples. Override the following methods for your task class: The harness is designed to facilitate task evaluations under the few-shot setting. Here we’ll format such examples. Override the following methods for your task class:
Put the natural language task description as a single line (no `\n`s) string here. E.g. `"Translate English to French:"`
```python ```python
def fewshot_description(self): def fewshot_description(self):
return "" return ""
``` ```
Put your natural language task description as a single line (no `\n`s) string here. E.g. `"Translate English to French:"`
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example (in dictionary form) . You should concatenate its members into a nicely formatted prompt.
```python ```python
def doc_to_text(self, doc): def doc_to_text(self, doc):
return "" return ""
``` ```
Format your document into a single query prompt __without the answer__ here. This method takes a single `doc` example (in dictionary form) . You should concatenate its members into a nicely formatted prompt.
Put the target answer of the prompt here, in the form: `" " + <answer>`.
```python ```python
def doc_to_target(self, doc): def doc_to_target(self, doc):
return "" return ""
``` ```
Put the target answer of the prompt here, in the form: `" " + <answer>`.
Understand that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍. Understand that the strings from `doc_to_text` and `doc_to_target` will be concatenated together to build up labeled examples in the k-shot setting where k > 0. Design with that in mind 👍.
...@@ -159,7 +165,8 @@ python -m scripts.write_out \ ...@@ -159,7 +165,8 @@ python -m scripts.write_out \
--num_examples N --num_examples N
``` ```
Open the file specified at the `--output_base_path <path>` and ensure it passes the eye test. Open the file specified at the `--output_base_path <path>` and ensure it passes
a simple eye test.
### The Evaluation ### The Evaluation
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment