GitLab

do per character loss aggregation for multiple choice tasks (similar to OAI's per token aggregation)

Now, if a run gets interrupted halfway, you can easily resume

# Conflicts:
#	lm_eval/models/gpt2.py
#	lm_eval/tasks/squad.py

- Extract evaluator (still needs work to clean up)
- Add tests for evaluator
- Fix all the things that break on the new tests
- Misc cleanup

In particular, the following assumptions are FALSE in general:
tokenize(context + continuation) = tokenize(context) + tokenize(continuation)
len(tokenize(context + continuation)) = len(tokenize(context)) + len(tokenize(continuation))
tokenize(context + continuation)[:len(tokenize(context))] = tokenize(context)

So we need to tip-toe around the problem by being careful with how we do it.

In particular, using Fast is not just for performance; while behavour of GPT2Tokenizer differs across Transformers 2 and 3, GPT2TokenizerFast doesn't.

It's ok if subclasses don't implement download, no-op default is ok