GitLab

# Conflicts:
#	lm_eval/tasks/pile.py

Nit on BoolQ

This is due to the change in the bits_per_byte calculation.

We can have fallback instead after @researcher2 implements it in best_download

Allow users to specify headqa subset

bits_per_byte was calculated as average per-byte loglikelihood, which would work if loglikelihood was base-2 log,
but it is natural log. To correct for that, bits_per_byte should be divided by math.log(2).

Also, it should be true that 2^bits_per_byte == byte_perplexity, which is true after the fix.

Add BLiMP

FIX wnli

Refactor LM organization for more reuse

Additional clean-up of LM Refactor

Pin datasets to 1.15.1 to fix RACE

Remove `truthfulqa` dependency on `t5` and bump version