GitLab

bits_per_byte was calculated as average per-byte loglikelihood, which would work if loglikelihood was base-2 log,
but it is natural log. To correct for that, bits_per_byte should be divided by math.log(2).

Also, it should be true that 2^bits_per_byte == byte_perplexity, which is true after the fix.

do per character loss aggregation for multiple choice tasks (similar to OAI's per token aggregation)

Now, if a run gets interrupted halfway, you can easily resume

# Conflicts:
#	lm_eval/models/gpt2.py
#	lm_eval/tasks/squad.py

- Extract evaluator (still needs work to clean up)
- Add tests for evaluator
- Fix all the things that break on the new tests
- Misc cleanup