GitLab

fixed the reference to the inner model

in case `isinstance(model, lm_eval.base.LM)` line 80 gives error, because `model` is not `str`. A simple inline if statement fixes this.

When `limit` is <1, limit represents the percentage of the total number of examples.
If it is >=1,  then it means the number of examples per task (only use this for testing).

This reverts commit 5fe7e2c0, reversing
changes made to 14b8ec7e.

- saves the doc idx by order in dataset
- saves the full ctx. we can turn this off it is too much memory
- saves the idx of the fewshot examples
- saves the source (test/val) of the fewshot examples
- TODO: We now need to return the pred/target from process_results
- TODO: Make it so override-ed process-resutls don't have to re-impl
this.

Turned off generation tasks for now. Changed process to look at the metrics. Only accuracy implemented.

Remove arguments from main.py.

Add "decontamination" prefix to ngrams arguments.