Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
task_table.md 70.7 KB
Newer Older
jon-tow's avatar
Cleanup `README.md` and package deps  
jon-tow committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268 
|                        Task Name                        |Train|Val|Test|Val/Test Docs|                                                                                     Metrics                                                                                     |
|---------------------------------------------------------|-----|---|----|------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|anagrams1                                                |     |✓  |    |        10000|acc                                                                                                                                                                              |
|anagrams2                                                |     |✓  |    |        10000|acc                                                                                                                                                                              |
|anli_r1                                                  |✓    |✓  |✓   |         1000|acc                                                                                                                                                                              |
|anli_r2                                                  |✓    |✓  |✓   |         1000|acc                                                                                                                                                                              |
|anli_r3                                                  |✓    |✓  |✓   |         1200|acc                                                                                                                                                                              |
|arc_challenge                                            |✓    |✓  |✓   |         1172|acc, acc_norm                                                                                                                                                                    |
|arc_easy                                                 |✓    |✓  |✓   |         2376|acc, acc_norm                                                                                                                                                                    |
|arithmetic_1dc                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_2da                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_2dm                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_2ds                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_3da                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_3ds                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_4da                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_4ds                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_5da                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|arithmetic_5ds                                           |     |✓  |    |         2000|acc                                                                                                                                                                              |
|blimp_adjunct_island                                     |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_anaphor_gender_agreement                           |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_anaphor_number_agreement                           |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_animate_subject_passive                            |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_animate_subject_trans                              |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_causative                                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_complex_NP_island                                  |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_coordinate_structure_constraint_complex_left_branch|     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_coordinate_structure_constraint_object_extraction  |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_1                        |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_2                        |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_irregular_1              |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_irregular_2              |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_with_adj_2               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_with_adj_irregular_1     |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_with_adj_irregular_2     |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_determiner_noun_agreement_with_adjective_1         |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_distractor_agreement_relational_noun               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_distractor_agreement_relative_clause               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_drop_argument                                      |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_ellipsis_n_bar_1                                   |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_ellipsis_n_bar_2                                   |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_existential_there_object_raising                   |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_existential_there_quantifiers_1                    |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_existential_there_quantifiers_2                    |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_existential_there_subject_raising                  |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_expletive_it_object_raising                        |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_inchoative                                         |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_intransitive                                       |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_irregular_past_participle_adjectives               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_irregular_past_participle_verbs                    |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_irregular_plural_subject_verb_agreement_1          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_irregular_plural_subject_verb_agreement_2          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_left_branch_island_echo_question                   |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_left_branch_island_simple_question                 |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_matrix_question_npi_licensor_present               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_npi_present_1                                      |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_npi_present_2                                      |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_only_npi_licensor_present                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_only_npi_scope                                     |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_passive_1                                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_passive_2                                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_c_command                              |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_case_1                                 |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_case_2                                 |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_domain_1                               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_domain_2                               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_domain_3                               |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_principle_A_reconstruction                         |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_regular_plural_subject_verb_agreement_1            |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_regular_plural_subject_verb_agreement_2            |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_sentential_negation_npi_licensor_present           |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_sentential_negation_npi_scope                      |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_sentential_subject_island                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_superlative_quantifiers_1                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_superlative_quantifiers_2                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_tough_vs_raising_1                                 |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_tough_vs_raising_2                                 |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_transitive                                         |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_island                                          |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_questions_object_gap                            |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_questions_subject_gap                           |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_questions_subject_gap_long_distance             |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_vs_that_no_gap                                  |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_vs_that_no_gap_long_distance                    |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_vs_that_with_gap                                |     |✓  |    |         1000|acc                                                                                                                                                                              |
|blimp_wh_vs_that_with_gap_long_distance                  |     |✓  |    |         1000|acc                                                                                                                                                                              |
|boolq                                                    |✓    |✓  |    |         3270|acc                                                                                                                                                                              |
|cb                                                       |✓    |✓  |    |           56|acc, f1                                                                                                                                                                          |
|cola                                                     |✓    |✓  |    |         1043|mcc                                                                                                                                                                              |
|copa                                                     |✓    |✓  |    |          100|acc                                                                                                                                                                              |
|coqa                                                     |✓    |✓  |    |          500|f1, em                                                                                                                                                                           |
|cycle_letters                                            |     |✓  |    |        10000|acc                                                                                                                                                                              |
|drop                                                     |✓    |✓  |    |         9536|em, f1                                                                                                                                                                           |
|ethics_cm                                                |✓    |   |✓   |         3885|acc                                                                                                                                                                              |
|ethics_deontology                                        |✓    |   |✓   |         3596|acc, em                                                                                                                                                                          |
|ethics_justice                                           |✓    |   |✓   |         2704|acc, em                                                                                                                                                                          |
|ethics_utilitarianism                                    |✓    |   |✓   |         4808|acc                                                                                                                                                                              |
|ethics_utilitarianism_original                           |     |   |✓   |         4808|acc                                                                                                                                                                              |
|ethics_virtue                                            |✓    |   |✓   |         4975|acc, em                                                                                                                                                                          |
|gsm8k                                                    |✓    |   |✓   |         1319|acc                                                                                                                                                                              |
|headqa                                                   |✓    |✓  |✓   |         2742|acc, acc_norm                                                                                                                                                                    |
|headqa_en                                                |✓    |✓  |✓   |         2742|acc, acc_norm                                                                                                                                                                    |
|headqa_es                                                |✓    |✓  |✓   |         2742|acc, acc_norm                                                                                                                                                                    |
|hellaswag                                                |✓    |✓  |    |        10042|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-abstract_algebra                           |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-anatomy                                    |     |✓  |✓   |          135|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-astronomy                                  |     |✓  |✓   |          152|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-business_ethics                            |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-clinical_knowledge                         |     |✓  |✓   |          265|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_biology                            |     |✓  |✓   |          144|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_chemistry                          |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_computer_science                   |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_mathematics                        |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_medicine                           |     |✓  |✓   |          173|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-college_physics                            |     |✓  |✓   |          102|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-computer_security                          |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-conceptual_physics                         |     |✓  |✓   |          235|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-econometrics                               |     |✓  |✓   |          114|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-electrical_engineering                     |     |✓  |✓   |          145|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-elementary_mathematics                     |     |✓  |✓   |          378|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-formal_logic                               |     |✓  |✓   |          126|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-global_facts                               |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_biology                        |     |✓  |✓   |          310|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_chemistry                      |     |✓  |✓   |          203|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_computer_science               |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_european_history               |     |✓  |✓   |          165|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_geography                      |     |✓  |✓   |          198|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_government_and_politics        |     |✓  |✓   |          193|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_macroeconomics                 |     |✓  |✓   |          390|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_mathematics                    |     |✓  |✓   |          270|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_microeconomics                 |     |✓  |✓   |          238|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_physics                        |     |✓  |✓   |          151|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_psychology                     |     |✓  |✓   |          545|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_statistics                     |     |✓  |✓   |          216|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_us_history                     |     |✓  |✓   |          204|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-high_school_world_history                  |     |✓  |✓   |          237|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-human_aging                                |     |✓  |✓   |          223|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-human_sexuality                            |     |✓  |✓   |          131|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-international_law                          |     |✓  |✓   |          121|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-jurisprudence                              |     |✓  |✓   |          108|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-logical_fallacies                          |     |✓  |✓   |          163|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-machine_learning                           |     |✓  |✓   |          112|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-management                                 |     |✓  |✓   |          103|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-marketing                                  |     |✓  |✓   |          234|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-medical_genetics                           |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-miscellaneous                              |     |✓  |✓   |          783|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-moral_disputes                             |     |✓  |✓   |          346|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-moral_scenarios                            |     |✓  |✓   |          895|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-nutrition                                  |     |✓  |✓   |          306|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-philosophy                                 |     |✓  |✓   |          311|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-prehistory                                 |     |✓  |✓   |          324|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-professional_accounting                    |     |✓  |✓   |          282|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-professional_law                           |     |✓  |✓   |         1534|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-professional_medicine                      |     |✓  |✓   |          272|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-professional_psychology                    |     |✓  |✓   |          612|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-public_relations                           |     |✓  |✓   |          110|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-security_studies                           |     |✓  |✓   |          245|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-sociology                                  |     |✓  |✓   |          201|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-us_foreign_policy                          |     |✓  |✓   |          100|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-virology                                   |     |✓  |✓   |          166|acc, acc_norm                                                                                                                                                                    |
|hendrycksTest-world_religions                            |     |✓  |✓   |          171|acc, acc_norm                                                                                                                                                                    |
|iwslt17-ar-en                                            |     |   |✓   |         1460|bleu, chrf, ter                                                                                                                                                                  |
|iwslt17-en-ar                                            |     |   |✓   |         1460|bleu, chrf, ter                                                                                                                                                                  |
|lambada_openai                                           |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_cloze                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_mt_de                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_mt_en                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_mt_es                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_mt_fr                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_openai_mt_it                                     |     |✓  |    |         5153|ppl, acc                                                                                                                                                                         |
|lambada_standard                                         |     |✓  |✓   |         5153|ppl, acc                                                                                                                                                                         |
|lambada_standard_cloze                                   |     |✓  |✓   |         5153|ppl, acc                                                                                                                                                                         |
|logiqa                                                   |✓    |✓  |✓   |          651|acc, acc_norm                                                                                                                                                                    |
|math_algebra                                             |✓    |   |✓   |         1187|acc                                                                                                                                                                              |
|math_asdiv                                               |     |✓  |    |         2305|acc                                                                                                                                                                              |
|math_counting_and_prob                                   |✓    |   |✓   |          474|acc                                                                                                                                                                              |
|math_geometry                                            |✓    |   |✓   |          479|acc                                                                                                                                                                              |
|math_intermediate_algebra                                |✓    |   |✓   |          903|acc                                                                                                                                                                              |
|math_num_theory                                          |✓    |   |✓   |          540|acc                                                                                                                                                                              |
|math_prealgebra                                          |✓    |   |✓   |          871|acc                                                                                                                                                                              |
|math_precalc                                             |✓    |   |✓   |          546|acc                                                                                                                                                                              |
|mathqa                                                   |✓    |✓  |✓   |         2985|acc, acc_norm                                                                                                                                                                    |
|mc_taco                                                  |     |✓  |✓   |         9442|f1, em                                                                                                                                                                           |
|mnli                                                     |✓    |✓  |    |         9815|acc                                                                                                                                                                              |
|mnli_mismatched                                          |✓    |✓  |    |         9832|acc                                                                                                                                                                              |
|mrpc                                                     |✓    |✓  |    |          408|acc, f1                                                                                                                                                                          |
|multirc                                                  |✓    |✓  |    |         4848|acc                                                                                                                                                                              |
|mutual                                                   |✓    |✓  |    |          886|r@1, r@2, mrr                                                                                                                                                                    |
|mutual_plus                                              |✓    |✓  |    |          886|r@1, r@2, mrr                                                                                                                                                                    |
|openbookqa                                               |✓    |✓  |✓   |          500|acc, acc_norm                                                                                                                                                                    |
|pile_arxiv                                               |     |✓  |✓   |         2407|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_bookcorpus2                                         |     |✓  |✓   |           28|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_books3                                              |     |✓  |✓   |          269|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_dm-mathematics                                      |     |✓  |✓   |         1922|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_enron                                               |     |✓  |✓   |         1010|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_europarl                                            |     |✓  |✓   |          157|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_freelaw                                             |     |✓  |✓   |         5101|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_github                                              |     |✓  |✓   |        18195|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_gutenberg                                           |     |✓  |✓   |           80|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_hackernews                                          |     |✓  |✓   |         1632|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_nih-exporter                                        |     |✓  |✓   |         1884|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_opensubtitles                                       |     |✓  |✓   |          642|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_openwebtext2                                        |     |✓  |✓   |        32925|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_philpapers                                          |     |✓  |✓   |           68|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_pile-cc                                             |     |✓  |✓   |        52790|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_pubmed-abstracts                                    |     |✓  |✓   |        29895|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_pubmed-central                                      |     |✓  |✓   |         5911|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_stackexchange                                       |     |✓  |✓   |        30378|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_ubuntu-irc                                          |     |✓  |✓   |           22|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_uspto                                               |     |✓  |✓   |        11415|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_wikipedia                                           |     |✓  |✓   |        17511|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|pile_youtubesubtitles                                    |     |✓  |✓   |          342|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|piqa                                                     |✓    |✓  |    |         1838|acc, acc_norm                                                                                                                                                                    |
|prost                                                    |     |   |✓   |        18736|acc, acc_norm                                                                                                                                                                    |
|pubmedqa                                                 |     |   |✓   |         1000|acc                                                                                                                                                                              |
|qa4mre_2011                                              |     |   |✓   |          120|acc, acc_norm                                                                                                                                                                    |
|qa4mre_2012                                              |     |   |✓   |          160|acc, acc_norm                                                                                                                                                                    |
|qa4mre_2013                                              |     |   |✓   |          284|acc, acc_norm                                                                                                                                                                    |
|qasper                                                   |✓    |✓  |    |         1764|f1_yesno, f1_abstractive                                                                                                                                                         |
|qnli                                                     |✓    |✓  |    |         5463|acc                                                                                                                                                                              |
|qqp                                                      |✓    |✓  |    |        40430|acc, f1                                                                                                                                                                          |
|race                                                     |✓    |✓  |✓   |         1045|acc                                                                                                                                                                              |
|random_insertion                                         |     |✓  |    |        10000|acc                                                                                                                                                                              |
|record                                                   |✓    |✓  |    |        10000|f1, em                                                                                                                                                                           |
|reversed_words                                           |     |✓  |    |        10000|acc                                                                                                                                                                              |
|rte                                                      |✓    |✓  |    |          277|acc                                                                                                                                                                              |
|sciq                                                     |✓    |✓  |✓   |         1000|acc, acc_norm                                                                                                                                                                    |
|squad2                                                   |✓    |✓  |    |        11873|exact, f1, HasAns_exact, HasAns_f1, NoAns_exact, NoAns_f1, best_exact, best_f1                                                                                                   |
|sst                                                      |✓    |✓  |    |          872|acc                                                                                                                                                                              |
|swag                                                     |✓    |✓  |    |        20006|acc, acc_norm                                                                                                                                                                    |
|triviaqa                                                 |✓    |✓  |    |        11313|acc                                                                                                                                                                              |
|truthfulqa_gen                                           |     |✓  |    |          817|bleurt_max, bleurt_acc, bleurt_diff, bleu_max, bleu_acc, bleu_diff, rouge1_max, rouge1_acc, rouge1_diff, rouge2_max, rouge2_acc, rouge2_diff, rougeL_max, rougeL_acc, rougeL_diff|
|truthfulqa_mc                                            |     |✓  |    |          817|mc1, mc2                                                                                                                                                                         |
|webqs                                                    |✓    |   |✓   |         2032|acc                                                                                                                                                                              |
|wic                                                      |✓    |✓  |    |          638|acc                                                                                                                                                                              |
|wikitext                                                 |✓    |✓  |✓   |           62|word_perplexity, byte_perplexity, bits_per_byte                                                                                                                                  |
|winogrande                                               |✓    |✓  |    |         1267|acc                                                                                                                                                                              |
|wmt14-en-fr                                              |     |   |✓   |         3003|bleu, chrf, ter                                                                                                                                                                  |
|wmt14-fr-en                                              |     |   |✓   |         3003|bleu, chrf, ter                                                                                                                                                                  |
|wmt16-de-en                                              |     |   |✓   |         2999|bleu, chrf, ter                                                                                                                                                                  |
|wmt16-en-de                                              |     |   |✓   |         2999|bleu, chrf, ter                                                                                                                                                                  |
|wmt16-en-ro                                              |     |   |✓   |         1999|bleu, chrf, ter                                                                                                                                                                  |
|wmt16-ro-en                                              |     |   |✓   |         1999|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-cs-en                                              |     |   |✓   |          664|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-de-en                                              |     |   |✓   |          785|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-de-fr                                              |     |   |✓   |         1619|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-cs                                              |     |   |✓   |         1418|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-de                                              |     |   |✓   |         1418|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-iu                                              |     |   |✓   |         2971|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-ja                                              |     |   |✓   |         1000|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-km                                              |     |   |✓   |         2320|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-pl                                              |     |   |✓   |         1000|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-ps                                              |     |   |✓   |         2719|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-ru                                              |     |   |✓   |         2002|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-ta                                              |     |   |✓   |         1000|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-en-zh                                              |     |   |✓   |         1418|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-fr-de                                              |     |   |✓   |         1619|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-iu-en                                              |     |   |✓   |         2971|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-ja-en                                              |     |   |✓   |          993|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-km-en                                              |     |   |✓   |         2320|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-pl-en                                              |     |   |✓   |         1001|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-ps-en                                              |     |   |✓   |         2719|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-ru-en                                              |     |   |✓   |          991|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-ta-en                                              |     |   |✓   |          997|bleu, chrf, ter                                                                                                                                                                  |
|wmt20-zh-en                                              |     |   |✓   |         2000|bleu, chrf, ter                                                                                                                                                                  |
|wnli                                                     |✓    |✓  |    |           71|acc                                                                                                                                                                              |
|wsc                                                      |✓    |✓  |    |          104|acc                                                                                                                                                                              |
|wsc273                                                   |     |   |✓   |          273|acc                                                                                                                                                                              |