GitLab

Pytest update

Update with new PR

Enable saving all the examples.

Pulling eval harness updates

Use eval-hackathon branch for installing prompt-source.

- saves the doc idx by order in dataset
- saves the full ctx. we can turn this off it is too much memory
- saves the idx of the fewshot examples
- saves the source (test/val) of the fewshot examples
- TODO: We now need to return the pred/target from process_results
- TODO: Make it so override-ed process-resutls don't have to re-impl
this.

Add `ROUGE` metric to `PromptSourceTask`

Updated the requests so that its easier to understand.

Updated doc: If the answer choices is empty, then it is generation; else ranked choice. This will be the canonical approach when using PS.

Add optional max length to generation.