GitLab

model_args should only be things that affect output of the model
therefore, stuff like batch size, device, etc shouldn't be in there

Now, if a run gets interrupted halfway, you can easily resume

# Conflicts:
#	lm_eval/models/gpt2.py
#	lm_eval/tasks/squad.py

In particular, the following assumptions are FALSE in general:
tokenize(context + continuation) = tokenize(context) + tokenize(continuation)
len(tokenize(context + continuation)) = len(tokenize(context)) + len(tokenize(continuation))
tokenize(context + continuation)[:len(tokenize(context))] = tokenize(context)

So we need to tip-toe around the problem by being careful with how we do it.

In particular, using Fast is not just for performance; while behavour of GPT2Tokenizer differs across Transformers 2 and 3, GPT2TokenizerFast doesn't.

TODO: still need to add `until` everywhere