GitLab

Merge branch 'big-refactor' of https://github.com/EleutherAI/lm-evaluation-harness into benchmark-scripts

[Refactor] Use console script

Remove condition to check for `winograd_schema`

Early stop bug of greedy_until (primary_until should be a list of str)

[Refactor] Migrate xwinograd tasks to yaml

[Refactor] Fix tests

I discovered that the accuracy of all models (e.g., llama7b, llama13b, starcoder) in the 'gsm8k-cot' task was 0%. After a thorough investigation, I realized that the generated text for each question was causing an early stop, preventing the 'regex_pattern' from finding any answers. This issue was caused by an incorrect assignment of the 'primary_until' variable in the 'greedy_until' function. Specifically, 'primary_until' should be a list of strings instead of a single string, as the 'stop_sequences' parameter in the 'stop_sequences_criteria' function requires a List[str]. Once I assigned 'primary_until' to '[until[0]]', the accuracy of llama7b in the 'gsm8k-cot' task increased to 1.67%.

[Refactor] Migrate webqs task to yaml

Fix dummy model not invoking super class constructor 

[Test] fix test_evaluator.py

[Refactor] typo

[Refactor] Minor changes