Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
47d8de5a
Commit
47d8de5a
authored
May 14, 2024
by
JessicaOjo
Browse files
restructure mmlu folder
parent
065d91bb
Changes
38
Show whitespace changes
Inline
Side-by-side
Showing
18 changed files
with
88 additions
and
0 deletions
+88
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_common_translate_yaml
...l/tasks/afrimmlu/translate/afrimmlu_common_translate_yaml
+35
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_amh.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_amh.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_eng.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_eng.yaml
+4
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ewe.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ewe.yaml
+4
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_fra.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_fra.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_hau.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_hau.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ibo.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ibo.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_kin.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_kin.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lin.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lin.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lug.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lug.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_orm.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_orm.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sna.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sna.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sot.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sot.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_twi.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_twi.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_wol.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_wol.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_xho.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_xho.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_yor.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_yor.yaml
+3
-0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_zul.yaml
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_zul.yaml
+3
-0
No files found.
lm_eval/tasks/afrimmlu/translate/afrimmlu_common_translate_yaml
0 → 100644
View file @
47d8de5a
group:
- mmlu
- afrimmlu_translate
task: null
dataset_path: masakhane/afrimmlu-translate-test
dataset_name: null
output_type: multiple_choice
test_split: test
doc_to_text: !function utils.doc_to_text
doc_to_target: "{{['A', 'B', 'C', 'D'].index(answer)}}"
doc_to_choice: !function utils.doc_to_choice
should_decontaminate: true
doc_to_decontamination_query: "Question: {{question}}\nAnswer:"
metric_list:
- metric: f1
aggregation: !function utils.weighted_f1_score
# aggregation: mean
average: weighted
hf_evaluate: true
higher_is_better: True
ignore_case: true
ignore_punctuation: true
regexes_to_ignore:
- ","
- "\\$"
- metric: acc
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
regexes_to_ignore:
- ","
- "\\$"
metadata:
version: 1.0
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_amh.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
amh
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_amh_translate
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_eng.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
eng
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_eng_translate
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ewe.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
ewe
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_ewe_translate
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_fra.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
fra
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_fra_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_hau.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
hau
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_hau_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_ibo.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
ibo
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_ibo_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_kin.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
kin
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_kin_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lin.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
lin
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_lin_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_lug.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
lug
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_lug_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_orm.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
orm
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_orm_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sna.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
sna
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_sna_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_sot.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
sot
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_sot_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_twi.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
twi
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_twi_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_wol.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
wol
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_wol_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_xho.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
xho
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_xho_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_yor.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
yor
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_yor_translate
\ No newline at end of file
lm_eval/tasks/afrimmlu/translate/afrimmlu_translate_zul.yaml
0 → 100644
View file @
47d8de5a
dataset_name
:
zul
include
:
afrimmlu_common_translate_yaml
task
:
afrimmlu_zul_translate
\ No newline at end of file
Prev
1
2
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment