GitLab

Refactor to remove generate and fix some bad tokenization.

In particular, the following assumptions are FALSE in general:
tokenize(context + continuation) = tokenize(context) + tokenize(continuation)
len(tokenize(context + continuation)) = len(tokenize(context)) + len(tokenize(continuation))
tokenize(context + continuation)[:len(tokenize(context))] = tokenize(context)

So we need to tip-toe around the problem by being careful with how we do it.

In particular, using Fast is not just for performance; while behavour of GPT2Tokenizer differs across Transformers 2 and 3, GPT2TokenizerFast doesn't.

# Conflicts:
#	write_out.py

Changing implementation of the —limit flag

This should allow the user to only do the first few eval tasks.

Add SAT Analogy dataset

Add SAT analogies dataset. Manual download needed. Checksums currently not verified, but hash is included as a comment.

Sync up to EAI

add rte text

Add Natural Questions

Add piqa and implement include_target for triviqa, storycloze, coqa

Adding mnli_mismatched

Update drop to be consistent with gpt3 paper