add arc_easy and arc_challenge generation variants

lm_eval/tasks/arc/README.md

@@ -16,6 +16,7 @@ a co-occurrence method fail to answer correctly) and an Easy Set of 5,197 questi
 
 Homepage: https://allenai.org/data/arc
 
+Note: The 0-shot generation variants are based on the LLama 3.2 [implementation](https://huggingface.co/datasets/meta-llama/Llama-3.2-3B-Instruct-evals/viewer/Llama-3.2-3B-Instruct-evals__arc_challenge__details).
 
 ### Citation
 
@@ -38,11 +39,14 @@ None.
 #### Tags
 
 * `ai2_arc`: Evaluates `arc_easy` and `arc_challenge`
+* `ai2_arc_generation`: Evaluates `arc_easy_generation` and `arc_challenge_generation`
 
 #### Tasks
 
 * `arc_easy`
 * `arc_challenge`
+* `arc_easy_generation`
+* `arc_challenge_generation`
 
 ### Checklist
 

lm_eval/tasks/arc/arc_challenge_chat.yaml → lm_eval/tasks/arc/arc_challenge_generation.yaml

 tag:
-  - llama
-task: arc_challenge_chat
+  - llama3_tasks
+  - ai2_arc_generation
+task: arc_challenge_generation
 dataset_path: allenai/ai2_arc
 dataset_name: ARC-Challenge
 output_type: generate_until

lm_eval/tasks/arc/arc_easy_generation.yaml

+tag:
+  - llama3_tasks
+  - ai2_arc_generation
+task: arc_east_generation
+dataset_path: allenai/ai2_arc
+dataset_name: ARC-Easy
+output_type: generate_until
+training_split: train
+validation_split: validation
+test_split: test
+fewshot_split: train
+doc_to_text: 'Given the following question and four candidate answers (A, B, C and D), choose the best answer.\nQuestion: {{question.strip()}}\nA. {{choices.text[0]}}\nB. {{choices.text[1]}}\nC. {{choices.text[2]}}{% if choices.text|length > 3 %}\nD. {{choices.text[3]}}{% endif %}\nYour response should end with "The best answer is [the_answer_letter]" where the [the_answer_letter] is one of A, B, C or D.'
+gen_prefix: 'The best answer is'
+fewshot_delimiter: "\n\n"
+doc_to_target: "{{ 'ABCD'[answerKey|int - 1] if answerKey|string in '1234' else answerKey }}"
+num_fewshot: 0
+generation_kwargs:
+  max_gen_toks: 100
+  until:
+    - "\n\n"
+    - "."
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+filter_list:
+  - name: remove_whitespace
+    filter:
+      - function: remove_whitespace
+      - function: take_first
+metadata:
+  version: 1.0