# llama-7B ## llama-7B_anli_0-shot.json | Task |Version|Metric|Value| |Stderr| |-------|------:|------|----:|---|-----:| |anli_r1| 0|acc |34.80|± | 1.51| |anli_r2| 0|acc |33.70|± | 1.50| |anli_r3| 0|acc |36.58|± | 1.39| ## llama-7B_arithmetic_5-shot.json | Task |Version|Metric|Value| |Stderr| |--------------|------:|------|----:|---|-----:| |arithmetic_1dc| 0|acc | 0|± | 0| |arithmetic_2da| 0|acc | 0|± | 0| |arithmetic_2dm| 0|acc | 0|± | 0| |arithmetic_2ds| 0|acc | 0|± | 0| |arithmetic_3da| 0|acc | 0|± | 0| |arithmetic_3ds| 0|acc | 0|± | 0| |arithmetic_4da| 0|acc | 0|± | 0| |arithmetic_4ds| 0|acc | 0|± | 0| |arithmetic_5da| 0|acc | 0|± | 0| |arithmetic_5ds| 0|acc | 0|± | 0| ## llama-7B_bbh_3-shot.json | Task |Version| Metric |Value| |Stderr| |------------------------------------------------|------:|---------------------|----:|---|-----:| |bigbench_causal_judgement | 0|multiple_choice_grade|48.42|± | 3.64| |bigbench_date_understanding | 0|multiple_choice_grade|62.06|± | 2.53| |bigbench_disambiguation_qa | 0|multiple_choice_grade|35.27|± | 2.98| |bigbench_dyck_languages | 0|multiple_choice_grade|15.40|± | 1.14| |bigbench_formal_fallacies_syllogisms_negation | 0|multiple_choice_grade|51.35|± | 0.42| |bigbench_geometric_shapes | 0|multiple_choice_grade|17.83|± | 2.02| | | |exact_str_match | 0.00|± | 0.00| |bigbench_hyperbaton | 0|multiple_choice_grade|49.51|± | 0.22| |bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|29.00|± | 2.03| |bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|24.57|± | 1.63| |bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|39.33|± | 2.83| |bigbench_movie_recommendation | 0|multiple_choice_grade|40.40|± | 2.20| |bigbench_navigate | 0|multiple_choice_grade|49.50|± | 1.58| |bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|34.60|± | 1.06| |bigbench_ruin_names | 0|multiple_choice_grade|29.91|± | 2.17| |bigbench_salient_translation_error_detection | 0|multiple_choice_grade|16.53|± | 1.18| |bigbench_snarks | 0|multiple_choice_grade|50.83|± | 3.73| |bigbench_sports_understanding | 0|multiple_choice_grade|50.00|± | 1.59| |bigbench_temporal_sequences | 0|multiple_choice_grade|27.20|± | 1.41| |bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|18.24|± | 1.09| |bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|13.71|± | 0.82| |bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|39.33|± | 2.83| ## llama-7B_blimp_0-shot.json | Task |Version|Metric|Value| |Stderr| |---------------------------------------------------------|------:|------|----:|---|-----:| |blimp_adjunct_island | 0|acc | 53.9|± | 1.58| |blimp_anaphor_gender_agreement | 0|acc | 44.8|± | 1.57| |blimp_anaphor_number_agreement | 0|acc | 65.9|± | 1.50| |blimp_animate_subject_passive | 0|acc | 62.6|± | 1.53| |blimp_animate_subject_trans | 0|acc | 76.1|± | 1.35| |blimp_causative | 0|acc | 50.8|± | 1.58| |blimp_complex_NP_island | 0|acc | 41.6|± | 1.56| |blimp_coordinate_structure_constraint_complex_left_branch| 0|acc | 68.2|± | 1.47| |blimp_coordinate_structure_constraint_object_extraction | 0|acc | 62.9|± | 1.53| |blimp_determiner_noun_agreement_1 | 0|acc | 63.6|± | 1.52| |blimp_determiner_noun_agreement_2 | 0|acc | 59.8|± | 1.55| |blimp_determiner_noun_agreement_irregular_1 | 0|acc | 57.2|± | 1.57| |blimp_determiner_noun_agreement_irregular_2 | 0|acc | 60.2|± | 1.55| |blimp_determiner_noun_agreement_with_adj_2 | 0|acc | 54.0|± | 1.58| |blimp_determiner_noun_agreement_with_adj_irregular_1 | 0|acc | 56.3|± | 1.57| |blimp_determiner_noun_agreement_with_adj_irregular_2 | 0|acc | 59.1|± | 1.56| |blimp_determiner_noun_agreement_with_adjective_1 | 0|acc | 57.7|± | 1.56| |blimp_distractor_agreement_relational_noun | 0|acc | 44.1|± | 1.57| |blimp_distractor_agreement_relative_clause | 0|acc | 31.4|± | 1.47| |blimp_drop_argument | 0|acc | 70.1|± | 1.45| |blimp_ellipsis_n_bar_1 | 0|acc | 66.8|± | 1.49| |blimp_ellipsis_n_bar_2 | 0|acc | 79.4|± | 1.28| |blimp_existential_there_object_raising | 0|acc | 78.8|± | 1.29| |blimp_existential_there_quantifiers_1 | 0|acc | 68.3|± | 1.47| |blimp_existential_there_quantifiers_2 | 0|acc | 67.4|± | 1.48| |blimp_existential_there_subject_raising | 0|acc | 69.6|± | 1.46| |blimp_expletive_it_object_raising | 0|acc | 65.9|± | 1.50| |blimp_inchoative | 0|acc | 42.0|± | 1.56| |blimp_intransitive | 0|acc | 59.2|± | 1.55| |blimp_irregular_past_participle_adjectives | 0|acc | 42.9|± | 1.57| |blimp_irregular_past_participle_verbs | 0|acc | 72.5|± | 1.41| |blimp_irregular_plural_subject_verb_agreement_1 | 0|acc | 65.3|± | 1.51| |blimp_irregular_plural_subject_verb_agreement_2 | 0|acc | 70.0|± | 1.45| |blimp_left_branch_island_echo_question | 0|acc | 83.5|± | 1.17| |blimp_left_branch_island_simple_question | 0|acc | 74.0|± | 1.39| |blimp_matrix_question_npi_licensor_present | 0|acc | 11.7|± | 1.02| |blimp_npi_present_1 | 0|acc | 53.4|± | 1.58| |blimp_npi_present_2 | 0|acc | 53.0|± | 1.58| |blimp_only_npi_licensor_present | 0|acc | 81.4|± | 1.23| |blimp_only_npi_scope | 0|acc | 26.6|± | 1.40| |blimp_passive_1 | 0|acc | 70.2|± | 1.45| |blimp_passive_2 | 0|acc | 70.3|± | 1.45| |blimp_principle_A_c_command | 0|acc | 39.0|± | 1.54| |blimp_principle_A_case_1 | 0|acc | 98.5|± | 0.38| |blimp_principle_A_case_2 | 0|acc | 55.4|± | 1.57| |blimp_principle_A_domain_1 | 0|acc | 96.2|± | 0.60| |blimp_principle_A_domain_2 | 0|acc | 64.6|± | 1.51| |blimp_principle_A_domain_3 | 0|acc | 50.1|± | 1.58| |blimp_principle_A_reconstruction | 0|acc | 67.3|± | 1.48| |blimp_regular_plural_subject_verb_agreement_1 | 0|acc | 64.5|± | 1.51| |blimp_regular_plural_subject_verb_agreement_2 | 0|acc | 70.5|± | 1.44| |blimp_sentential_negation_npi_licensor_present | 0|acc | 94.0|± | 0.75| |blimp_sentential_negation_npi_scope | 0|acc | 58.8|± | 1.56| |blimp_sentential_subject_island | 0|acc | 60.6|± | 1.55| |blimp_superlative_quantifiers_1 | 0|acc | 61.2|± | 1.54| |blimp_superlative_quantifiers_2 | 0|acc | 56.1|± | 1.57| |blimp_tough_vs_raising_1 | 0|acc | 29.8|± | 1.45| |blimp_tough_vs_raising_2 | 0|acc | 76.8|± | 1.34| |blimp_transitive | 0|acc | 69.8|± | 1.45| |blimp_wh_island | 0|acc | 27.5|± | 1.41| |blimp_wh_questions_object_gap | 0|acc | 67.0|± | 1.49| |blimp_wh_questions_subject_gap | 0|acc | 72.0|± | 1.42| |blimp_wh_questions_subject_gap_long_distance | 0|acc | 74.6|± | 1.38| |blimp_wh_vs_that_no_gap | 0|acc | 84.8|± | 1.14| |blimp_wh_vs_that_no_gap_long_distance | 0|acc | 81.2|± | 1.24| |blimp_wh_vs_that_with_gap | 0|acc | 23.9|± | 1.35| |blimp_wh_vs_that_with_gap_long_distance | 0|acc | 22.7|± | 1.33| ## llama-7B_common_sense_reasoning_0-shot.json | Task |Version| Metric |Value| |Stderr| |-------------|------:|--------|----:|---|-----:| |arc_challenge| 0|acc |38.23|± | 1.42| | | |acc_norm|41.38|± | 1.44| |arc_easy | 0|acc |67.38|± | 0.96| | | |acc_norm|52.48|± | 1.02| |boolq | 1|acc |73.06|± | 0.78| |copa | 0|acc |84.00|± | 3.68| |hellaswag | 0|acc |56.39|± | 0.49| | | |acc_norm|72.98|± | 0.44| |mc_taco | 0|em |11.26| | | | | |f1 |48.27| | | |openbookqa | 0|acc |28.20|± | 2.01| | | |acc_norm|42.40|± | 2.21| |piqa | 0|acc |78.18|± | 0.96| | | |acc_norm|77.42|± | 0.98| |prost | 0|acc |25.69|± | 0.32| | | |acc_norm|28.03|± | 0.33| |swag | 0|acc |55.47|± | 0.35| | | |acc_norm|66.87|± | 0.33| |winogrande | 0|acc |66.93|± | 1.32| |wsc273 | 0|acc |80.95|± | 2.38| ## llama-7B_glue_0-shot.json | Task |Version|Metric|Value| |Stderr| |---------------|------:|------|----:|---|-----:| |cola | 0|mcc | 0.00|± | 0.00| |mnli | 0|acc |34.40|± | 0.48| |mnli_mismatched| 0|acc |35.72|± | 0.48| |mrpc | 0|acc |68.38|± | 2.30| | | |f1 |81.22|± | 1.62| |qnli | 0|acc |49.57|± | 0.68| |qqp | 0|acc |36.84|± | 0.24| | | |f1 |53.81|± | 0.26| |rte | 0|acc |53.07|± | 3.00| |sst | 0|acc |52.98|± | 1.69| |wnli | 1|acc |56.34|± | 5.93| ## llama-7B_gsm8k_8-shot.json |Task |Version|Metric|Value| |Stderr| |-----|------:|------|----:|---|-----:| |gsm8k| 0|acc | 8.04|± | 0.75| ## llama-7B_human_alignment_0-shot.json | Task |Version| Metric | Value | |Stderr| |---------------------------------------|------:|---------------------|------:|---|-----:| |crows_pairs_english_age | 0|likelihood_difference| 594.23|± | 79.03| | | |pct_stereotype | 51.65|± | 5.27| |crows_pairs_english_autre | 0|likelihood_difference|1101.14|± |589.08| | | |pct_stereotype | 45.45|± | 15.75| |crows_pairs_english_disability | 0|likelihood_difference| 966.97|± |113.86| | | |pct_stereotype | 66.15|± | 5.91| |crows_pairs_english_gender | 0|likelihood_difference| 791.74|± | 55.02| | | |pct_stereotype | 53.12|± | 2.79| |crows_pairs_english_nationality | 0|likelihood_difference| 676.26|± | 58.69| | | |pct_stereotype | 53.70|± | 3.40| |crows_pairs_english_physical_appearance| 0|likelihood_difference| 451.26|± | 69.32| | | |pct_stereotype | 50.00|± | 5.93| |crows_pairs_english_race_color | 0|likelihood_difference| 624.65|± | 32.39| | | |pct_stereotype | 46.65|± | 2.22| |crows_pairs_english_religion | 0|likelihood_difference| 721.96|± | 75.92| | | |pct_stereotype | 66.67|± | 4.49| |crows_pairs_english_sexual_orientation | 0|likelihood_difference| 830.48|± | 84.28| | | |pct_stereotype | 62.37|± | 5.05| |crows_pairs_english_socioeconomic | 0|likelihood_difference| 640.16|± | 54.20| | | |pct_stereotype | 56.84|± | 3.60| |crows_pairs_french_age | 0|likelihood_difference|1193.96|± |153.77| | | |pct_stereotype | 35.56|± | 5.07| |crows_pairs_french_autre | 0|likelihood_difference| 751.20|± |209.58| | | |pct_stereotype | 61.54|± | 14.04| |crows_pairs_french_disability | 0|likelihood_difference|1014.77|± |139.07| | | |pct_stereotype | 42.42|± | 6.13| |crows_pairs_french_gender | 0|likelihood_difference|1179.90|± | 87.14| | | |pct_stereotype | 52.02|± | 2.79| |crows_pairs_french_nationality | 0|likelihood_difference|1041.65|± | 90.66| | | |pct_stereotype | 40.71|± | 3.09| |crows_pairs_french_physical_appearance | 0|likelihood_difference| 704.51|± | 94.84| | | |pct_stereotype | 55.56|± | 5.90| |crows_pairs_french_race_color | 0|likelihood_difference|1204.89|± | 73.32| | | |pct_stereotype | 43.26|± | 2.31| |crows_pairs_french_religion | 0|likelihood_difference| 958.53|± | 87.50| | | |pct_stereotype | 43.48|± | 4.64| |crows_pairs_french_sexual_orientation | 0|likelihood_difference| 760.58|± | 79.39| | | |pct_stereotype | 67.03|± | 4.96| |crows_pairs_french_socioeconomic | 0|likelihood_difference| 980.84|± |101.51| | | |pct_stereotype | 52.04|± | 3.58| |ethics_cm | 0|acc | 56.91|± | 0.79| |ethics_deontology | 0|acc | 50.58|± | 0.83| | | |em | 0.22| | | |ethics_justice | 0|acc | 49.96|± | 0.96| | | |em | 0.15| | | |ethics_utilitarianism | 0|acc | 49.81|± | 0.72| |ethics_utilitarianism_original | 0|acc | 95.86|± | 0.29| |ethics_virtue | 0|acc | 20.98|± | 0.58| | | |em | 0.00| | | |toxigen | 0|acc | 43.09|± | 1.62| | | |acc_norm | 43.19|± | 1.62| ## llama-7B_lambada_0-shot.json | Task |Version|Metric| Value | | Stderr | |----------------------|------:|------|---------:|---|--------:| |lambada_openai | 0|ppl |2817465.09|± |138319.09| | | |acc | 0.00|± | 0.00| |lambada_openai_cloze | 0|ppl | 255777.71|± | 11345.77| | | |acc | 0.04|± | 0.03| |lambada_openai_mt_de | 0|ppl |1805613.68|± | 97892.79| | | |acc | 0.00|± | 0.00| |lambada_openai_mt_en | 0|ppl |2817465.09|± |138319.09| | | |acc | 0.00|± | 0.00| |lambada_openai_mt_es | 0|ppl |3818890.45|± |197999.05| | | |acc | 0.00|± | 0.00| |lambada_openai_mt_fr | 0|ppl |2111186.12|± |111724.43| | | |acc | 0.00|± | 0.00| |lambada_openai_mt_it | 0|ppl |3653680.57|± |197082.99| | | |acc | 0.00|± | 0.00| |lambada_standard | 0|ppl |2460346.86|± | 81216.57| | | |acc | 0.00|± | 0.00| |lambada_standard_cloze| 0|ppl |6710057.24|± |169833.91| | | |acc | 0.00|± | 0.00| ## llama-7B_mathematical_reasoning_0-shot.json | Task |Version| Metric |Value| |Stderr| |-------------------------|------:|--------|----:|---|-----:| |drop | 1|em | 4.27|± | 0.21| | | |f1 |12.16|± | 0.25| |gsm8k | 0|acc | 0.00|± | 0.00| |math_algebra | 1|acc | 1.68|± | 0.37| |math_asdiv | 0|acc | 0.00|± | 0.00| |math_counting_and_prob | 1|acc | 1.69|± | 0.59| |math_geometry | 1|acc | 0.84|± | 0.42| |math_intermediate_algebra| 1|acc | 0.66|± | 0.27| |math_num_theory | 1|acc | 0.74|± | 0.37| |math_prealgebra | 1|acc | 1.26|± | 0.38| |math_precalc | 1|acc | 0.37|± | 0.26| |mathqa | 0|acc |26.77|± | 0.81| | | |acc_norm|27.87|± | 0.82| ## llama-7B_mathematical_reasoning_few_shot_5-shot.json | Task |Version| Metric |Value| |Stderr| |-------------------------|------:|--------|----:|---|-----:| |drop | 1|em | 1.24|± | 0.11| | | |f1 | 2.10|± | 0.13| |gsm8k | 0|acc | 0.00|± | 0.00| |math_algebra | 1|acc | 0.00|± | 0.00| |math_counting_and_prob | 1|acc | 0.00|± | 0.00| |math_geometry | 1|acc | 0.00|± | 0.00| |math_intermediate_algebra| 1|acc | 0.00|± | 0.00| |math_num_theory | 1|acc | 0.00|± | 0.00| |math_prealgebra | 1|acc | 0.11|± | 0.11| |math_precalc | 1|acc | 0.00|± | 0.00| |mathqa | 0|acc |28.21|± | 0.82| | | |acc_norm|28.78|± | 0.83| ## llama-7B_mmlu_5-shot.json | Task |Version| Metric |Value| |Stderr| |-------------------------------------------------|------:|--------|----:|---|-----:| |hendrycksTest-abstract_algebra | 0|acc |23.00|± | 4.23| | | |acc_norm|26.00|± | 4.41| |hendrycksTest-anatomy | 0|acc |38.52|± | 4.20| | | |acc_norm|28.15|± | 3.89| |hendrycksTest-astronomy | 0|acc |45.39|± | 4.05| | | |acc_norm|46.05|± | 4.06| |hendrycksTest-business_ethics | 0|acc |53.00|± | 5.02| | | |acc_norm|46.00|± | 5.01| |hendrycksTest-clinical_knowledge | 0|acc |38.87|± | 3.00| | | |acc_norm|38.11|± | 2.99| |hendrycksTest-college_biology | 0|acc |31.94|± | 3.90| | | |acc_norm|29.17|± | 3.80| |hendrycksTest-college_chemistry | 0|acc |33.00|± | 4.73| | | |acc_norm|30.00|± | 4.61| |hendrycksTest-college_computer_science | 0|acc |33.00|± | 4.73| | | |acc_norm|28.00|± | 4.51| |hendrycksTest-college_mathematics | 0|acc |32.00|± | 4.69| | | |acc_norm|32.00|± | 4.69| |hendrycksTest-college_medicine | 0|acc |37.57|± | 3.69| | | |acc_norm|30.64|± | 3.51| |hendrycksTest-college_physics | 0|acc |23.53|± | 4.22| | | |acc_norm|32.35|± | 4.66| |hendrycksTest-computer_security | 0|acc |37.00|± | 4.85| | | |acc_norm|44.00|± | 4.99| |hendrycksTest-conceptual_physics | 0|acc |32.77|± | 3.07| | | |acc_norm|21.70|± | 2.69| |hendrycksTest-econometrics | 0|acc |28.95|± | 4.27| | | |acc_norm|26.32|± | 4.14| |hendrycksTest-electrical_engineering | 0|acc |35.86|± | 4.00| | | |acc_norm|32.41|± | 3.90| |hendrycksTest-elementary_mathematics | 0|acc |32.01|± | 2.40| | | |acc_norm|29.10|± | 2.34| |hendrycksTest-formal_logic | 0|acc |30.95|± | 4.13| | | |acc_norm|34.92|± | 4.26| |hendrycksTest-global_facts | 0|acc |32.00|± | 4.69| | | |acc_norm|29.00|± | 4.56| |hendrycksTest-high_school_biology | 0|acc |35.81|± | 2.73| | | |acc_norm|35.81|± | 2.73| |hendrycksTest-high_school_chemistry | 0|acc |25.12|± | 3.05| | | |acc_norm|29.56|± | 3.21| |hendrycksTest-high_school_computer_science | 0|acc |41.00|± | 4.94| | | |acc_norm|34.00|± | 4.76| |hendrycksTest-high_school_european_history | 0|acc |40.61|± | 3.83| | | |acc_norm|36.97|± | 3.77| |hendrycksTest-high_school_geography | 0|acc |42.93|± | 3.53| | | |acc_norm|36.36|± | 3.43| |hendrycksTest-high_school_government_and_politics| 0|acc |48.19|± | 3.61| | | |acc_norm|37.31|± | 3.49| |hendrycksTest-high_school_macroeconomics | 0|acc |31.79|± | 2.36| | | |acc_norm|30.26|± | 2.33| |hendrycksTest-high_school_mathematics | 0|acc |22.59|± | 2.55| | | |acc_norm|30.74|± | 2.81| |hendrycksTest-high_school_microeconomics | 0|acc |38.66|± | 3.16| | | |acc_norm|36.55|± | 3.13| |hendrycksTest-high_school_physics | 0|acc |20.53|± | 3.30| | | |acc_norm|27.15|± | 3.63| |hendrycksTest-high_school_psychology | 0|acc |46.61|± | 2.14| | | |acc_norm|30.83|± | 1.98| |hendrycksTest-high_school_statistics | 0|acc |34.26|± | 3.24| | | |acc_norm|34.26|± | 3.24| |hendrycksTest-high_school_us_history | 0|acc |42.65|± | 3.47| | | |acc_norm|31.37|± | 3.26| |hendrycksTest-high_school_world_history | 0|acc |39.24|± | 3.18| | | |acc_norm|33.76|± | 3.08| |hendrycksTest-human_aging | 0|acc |37.22|± | 3.24| | | |acc_norm|25.11|± | 2.91| |hendrycksTest-human_sexuality | 0|acc |51.15|± | 4.38| | | |acc_norm|36.64|± | 4.23| |hendrycksTest-international_law | 0|acc |38.84|± | 4.45| | | |acc_norm|57.85|± | 4.51| |hendrycksTest-jurisprudence | 0|acc |43.52|± | 4.79| | | |acc_norm|50.00|± | 4.83| |hendrycksTest-logical_fallacies | 0|acc |38.04|± | 3.81| | | |acc_norm|34.97|± | 3.75| |hendrycksTest-machine_learning | 0|acc |30.36|± | 4.36| | | |acc_norm|26.79|± | 4.20| |hendrycksTest-management | 0|acc |48.54|± | 4.95| | | |acc_norm|36.89|± | 4.78| |hendrycksTest-marketing | 0|acc |61.11|± | 3.19| | | |acc_norm|50.43|± | 3.28| |hendrycksTest-medical_genetics | 0|acc |44.00|± | 4.99| | | |acc_norm|40.00|± | 4.92| |hendrycksTest-miscellaneous | 0|acc |58.37|± | 1.76| | | |acc_norm|38.95|± | 1.74| |hendrycksTest-moral_disputes | 0|acc |36.42|± | 2.59| | | |acc_norm|33.24|± | 2.54| |hendrycksTest-moral_scenarios | 0|acc |27.60|± | 1.50| | | |acc_norm|27.26|± | 1.49| |hendrycksTest-nutrition | 0|acc |39.54|± | 2.80| | | |acc_norm|43.79|± | 2.84| |hendrycksTest-philosophy | 0|acc |40.19|± | 2.78| | | |acc_norm|35.37|± | 2.72| |hendrycksTest-prehistory | 0|acc |40.12|± | 2.73| | | |acc_norm|27.78|± | 2.49| |hendrycksTest-professional_accounting | 0|acc |30.14|± | 2.74| | | |acc_norm|29.43|± | 2.72| |hendrycksTest-professional_law | 0|acc |29.66|± | 1.17| | | |acc_norm|28.55|± | 1.15| |hendrycksTest-professional_medicine | 0|acc |33.82|± | 2.87| | | |acc_norm|27.94|± | 2.73| |hendrycksTest-professional_psychology | 0|acc |38.40|± | 1.97| | | |acc_norm|29.90|± | 1.85| |hendrycksTest-public_relations | 0|acc |39.09|± | 4.67| | | |acc_norm|22.73|± | 4.01| |hendrycksTest-security_studies | 0|acc |40.82|± | 3.15| | | |acc_norm|31.02|± | 2.96| |hendrycksTest-sociology | 0|acc |47.76|± | 3.53| | | |acc_norm|42.79|± | 3.50| |hendrycksTest-us_foreign_policy | 0|acc |56.00|± | 4.99| | | |acc_norm|45.00|± | 5.00| |hendrycksTest-virology | 0|acc |39.76|± | 3.81| | | |acc_norm|28.92|± | 3.53| |hendrycksTest-world_religions | 0|acc |62.57|± | 3.71| | | |acc_norm|51.46|± | 3.83| ## llama-7B_pawsx_0-shot.json | Task |Version|Metric|Value| |Stderr| |--------|------:|------|----:|---|-----:| |pawsx_de| 0|acc |54.65|± | 1.11| |pawsx_en| 0|acc |61.85|± | 1.09| |pawsx_es| 0|acc |56.10|± | 1.11| |pawsx_fr| 0|acc |52.95|± | 1.12| |pawsx_ja| 0|acc |56.70|± | 1.11| |pawsx_ko| 0|acc |49.70|± | 1.12| |pawsx_zh| 0|acc |49.10|± | 1.12| ## llama-7B_question_answering_0-shot.json | Task |Version| Metric |Value| |Stderr| |-------------|------:|------------|----:|---|-----:| |headqa_en | 0|acc |32.42|± | 0.89| | | |acc_norm |35.92|± | 0.92| |headqa_es | 0|acc |28.26|± | 0.86| | | |acc_norm |32.42|± | 0.89| |logiqa | 0|acc |21.81|± | 1.62| | | |acc_norm |30.26|± | 1.80| |squad2 | 1|exact | 9.42| | | | | |f1 |19.45| | | | | |HasAns_exact|18.49| | | | | |HasAns_f1 |38.58| | | | | |NoAns_exact | 0.37| | | | | |NoAns_f1 | 0.37| | | | | |best_exact |50.07| | | | | |best_f1 |50.08| | | |triviaqa | 1|acc | 0.00|± | 0.00| |truthfulqa_mc| 1|mc1 |21.05|± | 1.43| | | |mc2 |34.14|± | 1.31| |webqs | 0|acc | 0.00|± | 0.00| ## llama-7B_reading_comprehension_0-shot.json |Task|Version|Metric|Value| |Stderr| |----|------:|------|----:|---|-----:| |coqa| 1|f1 |75.21|± | 1.53| | | |em |62.67|± | 1.88| |drop| 1|em | 3.59|± | 0.19| | | |f1 |11.35|± | 0.23| |race| 1|acc |39.90|± | 1.52| ## llama-7B_unscramble_0-shot.json | Task |Version|Metric|Value| |Stderr| |----------------|------:|------|----:|---|-----:| |anagrams1 | 0|acc | 0|± | 0| |anagrams2 | 0|acc | 0|± | 0| |cycle_letters | 0|acc | 0|± | 0| |random_insertion| 0|acc | 0|± | 0| |reversed_words | 0|acc | 0|± | 0| ## llama-7B_xcopa_0-shot.json | Task |Version|Metric|Value| |Stderr| |--------|------:|------|----:|---|-----:| |xcopa_et| 0|acc | 48.8|± | 2.24| |xcopa_ht| 0|acc | 51.0|± | 2.24| |xcopa_id| 0|acc | 54.6|± | 2.23| |xcopa_it| 0|acc | 62.0|± | 2.17| |xcopa_qu| 0|acc | 51.4|± | 2.24| |xcopa_sw| 0|acc | 50.8|± | 2.24| |xcopa_ta| 0|acc | 55.2|± | 2.23| |xcopa_th| 0|acc | 55.8|± | 2.22| |xcopa_tr| 0|acc | 55.6|± | 2.22| |xcopa_vi| 0|acc | 51.6|± | 2.24| |xcopa_zh| 0|acc | 56.2|± | 2.22| ## llama-7B_xnli_0-shot.json | Task |Version|Metric|Value| |Stderr| |-------|------:|------|----:|---|-----:| |xnli_ar| 0|acc |33.57|± | 0.67| |xnli_bg| 0|acc |36.99|± | 0.68| |xnli_de| 0|acc |44.77|± | 0.70| |xnli_el| 0|acc |34.93|± | 0.67| |xnli_en| 0|acc |51.06|± | 0.71| |xnli_es| 0|acc |40.62|± | 0.69| |xnli_fr| 0|acc |43.75|± | 0.70| |xnli_hi| 0|acc |36.11|± | 0.68| |xnli_ru| 0|acc |39.36|± | 0.69| |xnli_sw| 0|acc |33.71|± | 0.67| |xnli_th| 0|acc |34.51|± | 0.67| |xnli_tr| 0|acc |35.59|± | 0.68| |xnli_ur| 0|acc |33.39|± | 0.67| |xnli_vi| 0|acc |35.59|± | 0.68| |xnli_zh| 0|acc |36.23|± | 0.68| ## llama-7B_xstory_cloze_0-shot.json | Task |Version|Metric|Value| |Stderr| |---------------|------:|------|----:|---|-----:| |xstory_cloze_ar| 0|acc |48.31|± | 1.29| |xstory_cloze_en| 0|acc |74.78|± | 1.12| |xstory_cloze_es| 0|acc |65.12|± | 1.23| |xstory_cloze_eu| 0|acc |50.10|± | 1.29| |xstory_cloze_hi| 0|acc |52.68|± | 1.28| |xstory_cloze_id| 0|acc |52.08|± | 1.29| |xstory_cloze_my| 0|acc |48.71|± | 1.29| |xstory_cloze_ru| 0|acc |61.35|± | 1.25| |xstory_cloze_sw| 0|acc |50.36|± | 1.29| |xstory_cloze_te| 0|acc |52.88|± | 1.28| |xstory_cloze_zh| 0|acc |54.33|± | 1.28| ## llama-7B_xwinograd_0-shot.json | Task |Version|Metric|Value| |Stderr| |------------|------:|------|----:|---|-----:| |xwinograd_en| 0|acc |84.95|± | 0.74| |xwinograd_fr| 0|acc |72.29|± | 4.94| |xwinograd_jp| 0|acc |58.92|± | 1.59| |xwinograd_pt| 0|acc |70.72|± | 2.81| |xwinograd_ru| 0|acc |64.44|± | 2.70| |xwinograd_zh| 0|acc |63.69|± | 2.14|