GitLab

request construction

This reverts commit 5fe7e2c0, reversing
changes made to 14b8ec7e.

- saves the doc idx by order in dataset
- saves the full ctx. we can turn this off it is too much memory
- saves the idx of the fewshot examples
- saves the source (test/val) of the fewshot examples
- TODO: We now need to return the pred/target from process_results
- TODO: Make it so override-ed process-resutls don't have to re-impl
this.

Updated doc: If the answer choices is empty, then it is generation; else ranked choice. This will be the canonical approach when using PS.

Turned off generation tasks for now. Changed process to look at the metrics. Only accuracy implemented.