added additional squad tasks

4578ca14 · lintangsutawika · 1ed76cfa · 4578ca14 · 4578ca14
Commit 4578ca14 authored Sep 19, 2023 by lintangsutawika
Hide whitespace changes
Inline Side-by-side

Showing with 13 additions and 0 deletions

lm_eval/tasks/squadv2/no_ans.yaml lm_eval/tasks/squadv2/no_ans.yaml +9 -0

lm_eval/tasks/squadv2/with_noans_prob.yaml lm_eval/tasks/squadv2/with_noans_prob.yaml +4 -0

No files found.
--- a/lm_eval/tasks/squadv2/no_ans.yaml
+++ b/lm_eval/tasks/squadv2/no_ans.yaml
+include: default.yaml
+task: squadv2_noans_loglikelihood
+dataset_path: squad_v2
+output_type: loglikelihood
+training_split: train
+validation_split: validation
+doc_to_target: " unanswerable"
+metric_list:
+  - metric: perplexity
--- a/lm_eval/tasks/squadv2/with_noans_prob.yaml
+++ b/lm_eval/tasks/squadv2/with_noans_prob.yaml
+group: squadv2_complete
+task:
+  - squadv2
+  - squadv2_noans_loglikelihood
\ No newline at end of file