GitLab

* Fix tokenization issue in BaseLM.loglikelihood

* Add a regression script

* Use entire non-continuation length as context

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

When `limit` is <1, limit represents the percentage of the total number of examples.
If it is >=1,  then it means the number of examples per task (only use this for testing).

This makes comparing the results of different models easier because tasks are ordered in the same way.

This reverts commit 5fe7e2c0, reversing
changes made to 14b8ec7e.

- saves the doc idx by order in dataset
- saves the full ctx. we can turn this off it is too much memory
- saves the idx of the fewshot examples
- saves the source (test/val) of the fewshot examples
- TODO: We now need to return the pred/target from process_results
- TODO: Make it so override-ed process-resutls don't have to re-impl
this.

Remove arguments from main.py.

Add "decontamination" prefix to ngrams arguments.

for some reason putting it in LM and having it be inherited breaks
everything. should try to figure this out at some point.

model_args should only be things that affect output of the model
therefore, stuff like batch size, device, etc shouldn't be in there