Merge pull request #6 from JessicaOjo/africanli

update metrics for afrixnli

lm_eval/tasks/afrixnli/en-direct/afrixnli_en_direct_yaml

@@ -20,15 +20,11 @@ should_decontaminate: true
 doc_to_decontamination_query: premise
 metric_list:
   - metric: f1
-    aggregation: !function utils.weighted_f1_score
+    aggregation: f1
     average: weighted
-    hf_evaluate: true
     higher_is_better: True
     ignore_case: true
     ignore_punctuation: true
-    regexes_to_ignore:
-      - ","
-      - "\\$"
   - metric: acc
     aggregation: mean
     higher_is_better: true

lm_eval/tasks/afrixnli/native-direct/afrixnli_native_direct_yaml

@@ -16,16 +16,10 @@ metric_list:
     higher_is_better: True
     ignore_case: true
     ignore_punctuation: true
-    regexes_to_ignore:
-      - ","
-      - "\\$"
   - metric: acc
     aggregation: mean
     higher_is_better: true
     ignore_case: true
     ignore_punctuation: true
-    regexes_to_ignore:
-      - ","
-      - "\\$"
 metadata:
   version: 1.0

results/afrixnli_samples/afrixnli_en_direct_amh

 !!@@##@@!! -- Example 0
-እንደ ተግዳሮቶች በአሁኑ ወቅት ወደ  ሁለት ተከታታይ ትናንሽ ቅርጾች ሊቆረጥ የሚችል ቃል እየፈለግሁ ነው።
-Question: ቀላል ጊዜ አልነበረኝም ለሁለት ተከታታይ ትናንሽ ቅርጾች ሊቆራረጥ የሚችል ቃል ለማግኘት ። True, False, or Neither?
-Answer: True
+ስናየው በተወሰነ ደረጃ ፈርተን ነበር፣ ነገር ግን ያለ ጕጕት በልተነዋል፣ ነገር ግን በቆሰለ የላይኛው ከንፈር አማካኝነት ለመናገር ያህል ጠጥተናል የእናታችንን ወተት።
+Question: ሁሉም ሰው ምግባቸውን በጋለ ስሜት በሉና የሚወዷቸውን የበኣል ምግቦች እንዳስታወሶቸው አስተያየት ሰጡ። True, False, or Neither?
+Answer: False
 
 እንግዲህ , እኔ ስለ እሱ እያሰብኩኝ አልነበረም , ነገር ግን እኔ በጣም ተስፋ ቆርጨ ነበር, እና, እኔ እንደገና ከእርሱ ጋር ማውራት ጀመርኩ።
 Question: እኔ ከእርሱ ጋር ድጋሜ አልተነጋገርኩም። True, False, or Neither?
 Answer:
 !!@@##@@!! -- Example 1
-ስጦታህ ለ85ኛ ዓመት ክብረ በዓላችን አስፈላጊ ነው።
-Question: ያገኘነው ስጦታ ሁሉ እንደ አንተ ስጦታ አስፈላጊ አይደለም። True, False, or Neither?
-Answer: Neither
+መሄድ አልተፈቀደለትም
+Question: ለመጀመሪያ ጊዜ የተጋበዘ እና በልምዱ የተደሰተ ነው True, False, or Neither?
+Answer: False
 
 እንግዲህ , እኔ ስለ እሱ እያሰብኩኝ አልነበረም , ነገር ግን እኔ በጣም ተስፋ ቆርጨ ነበር, እና, እኔ እንደገና ከእርሱ ጋር ማውራት ጀመርኩ።
 Question: እኔ በጣም ስለተበሳጨሁ እንደገና እሱን ማነጋገር ጀመርኩ። True, False, or Neither?
 Answer:
 !!@@##@@!! -- Example 2
-በንዑስ ክፍል አቀራረብ ውስጥ መሰረታዊ እና የሥራ-ድርሻ ምድብ እያንዳንዳቸው አማካይ መጠናቸውን ለማግኘት ከዋጋው በላይ መቶኛ ምልክት ይሰጣቸዋል።
-Question: ዋጋው ከወጪው 10% የበለጠ ነው፡፡ True, False, or Neither?
+.., ግን ለሁለተኛው ጊዜ ግንኙነት በሁለት ጓደኞች መካከል ላይ ሆነ።
+Question: ከሁለቱ ጓደኞቹ መካከል የትኛውን ወገን መምረጥ እንዳለበት አያውቅም። True, False, or Neither?
 Answer: Neither
 
 እንግዲህ , እኔ ስለ እሱ እያሰብኩኝ አልነበረም , ነገር ግን እኔ በጣም ተስፋ ቆርጨ ነበር, እና, እኔ እንደገና ከእርሱ ጋር ማውራት ጀመርኩ።

results/afrixnli_samples/afrixnli_en_direct_eng

 !!@@##@@!! -- Example 0
-When civil society turns a deaf ear, crazy ideas lose their edge.
-Question: Crazy ideas become more popular with civil society when they are ignored by civil society. True, False, or Neither?
+good brother -- this expression of common present usage is found in Julius Caesar (iv.
+Question: The term good brother has only been used since the 20th Century. True, False, or Neither?
 Answer: False
 
 Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again.
 Question: I havent spoken to him again. True, False, or Neither?
 Answer:
 !!@@##@@!! -- Example 1
-The shortage of skilled IT workers in the current market environment is often a major reason for leading organizations to outsource.
-Question: There aren't enough IT workers because they all went to India. True, False, or Neither?
-Answer: Neither
+Delivering it repeatedly did not make it more effective.
+Question: Delivery did not improve how effective it was. True, False, or Neither?
+Answer: True
 
 Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again.
 Question: I was so upset that I just started talking to him again. True, False, or Neither?
 Answer:
 !!@@##@@!! -- Example 2
-yeah you you must've had a cordless
-Question: You might have had one with an optional cord. True, False, or Neither?
-Answer: Neither
+Well, given that Bill Bradley grew up in St. Louis   wait, sorry, that would only be funny if Al Gore had grown up in Tennessee.
+Question: Bradley was from Arkansas. True, False, or Neither?
+Answer: False
 
 Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again.
 Question: We had a great talk. True, False, or Neither?