GitLab

do per character loss aggregation for multiple choice tasks (similar to OAI's per token aggregation)

(or rather, move it to the data download phase)

Now, if a run gets interrupted halfway, you can easily resume