navigate.yaml

"dataset_name": "navigate"
"description": "Given a series of navigation instructions, determine whether one would end up back at the starting point.\n\n"
"doc_to_text": "Q: If you follow these instructions, do you return to the starting point? Turn left. Turn around. Turn left. Take 7 steps. Take 2 steps. Take 4 steps. Take 8 steps.\nOptions:\n- Yes\n- No\nA: No\n\nQ: If you follow these instructions, do you return to the starting point? Turn around. Take 1 step. Take 6 steps. Turn around. Take 6 steps. Take 9 steps. Take 1 step.\nOptions:\n- Yes\n- No\nA: No\n\nQ: If you follow these instructions, do you return to the starting point? Always face forward. Take 2 steps right. Take 9 steps left. Take 7 steps right.\nOptions:\n- Yes\n- No\nA: Yes\n\nQ: {{input}}\nA:"
"include": "_fewshot_template_yaml"
"task": "bbh_fewshot_navigate"
filter_list:
  - name: "strict-match"
    filter:
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: 0
        regex_pattern: "\\b(Yes|No|yes|no)\\b"
      - function: "take_first"