Finetuning Models on Downstream Tasks

The GPT-3 paper didn't explore fine tuning on downstream tasks, so I decided to tune Neo 2.7B for 1.1k iters on all the tasks in eval harness that have a train set (all at once, because tuning one model per task would have taken ages). I was quite surprised that the tuned model didn't destroy untuned 2.7B completely on all tasks, but rather from eyeballing it seems like a tossup. Interestingly, tuned seems to defeat 2.7B by quite a lot on anli, which is especially notable given that this is one task the models in the GPT-3 paper struggled on. Also, lambada and pubmedqa are included in these tables, even though it doesn't have a training set (at least for the implementation in eval harness, using the OA version of lambada), because I wanted to look at effects on sets not in the tuning, to potentially observe some catastrophic forgetting or something. Sure enough, lambada and pubmedqa scores are significantly worse on the tuned model.

Zero shot

Task Metric 2.7B Tuned

anli_r1 acc 0.332 ± 0.015 0.418 ± 0.015

anli_r2 acc 0.342 ± 0.015 0.375 ± 0.015

anli_r3 acc 0.352 ± 0.014 0.392 ± 0.014

arc_challenge acc 0.275 ± 0.013 0.286 ± 0.013

acc_norm 0.301 ± 0.013 0.312 ± 0.013

arc_easy acc 0.611 ± 0.010 0.560 ± 0.010

acc_norm 0.539 ± 0.010 0.558 ± 0.010

boolq acc 0.630 ± 0.008 0.605 ± 0.008

cb acc 0.304 ± 0.062 0.411 ± 0.062

copa acc 0.800 ± 0.040 0.730 ± 0.040

ethics_cm acc 0.510 ± 0.008 0.561 ± 0.008

ethics_deontology acc 0.497 ± 0.008 0.658 ± 0.008

ethics_justice acc 0.501 ± 0.010 0.589 ± 0.010

ethics_utilitarianism acc 0.497 ± 0.007 0.498 ± 0.007

ethics_virtue acc 0.251 ± 0.006 0.800 ± 0.006

headqa acc 0.235 ± 0.008 0.233 ± 0.008

acc_norm 0.272 ± 0.008 0.265 ± 0.008

hellaswag acc 0.427 ± 0.005 0.400 ± 0.005

acc_norm 0.558 ± 0.005 0.517 ± 0.005

hendrycksTest-abstract_algebra acc 0.230 ± 0.042 0.340 ± 0.042

acc_norm 0.200 ± 0.040 0.350 ± 0.040

hendrycksTest-anatomy acc 0.252 ± 0.037 0.267 ± 0.037

acc_norm 0.222 ± 0.036 0.252 ± 0.036

hendrycksTest-astronomy acc 0.250 ± 0.035 0.309 ± 0.035

acc_norm 0.362 ± 0.039 0.309 ± 0.039

hendrycksTest-business_ethics acc 0.360 ± 0.048 0.340 ± 0.048

acc_norm 0.280 ± 0.045 0.310 ± 0.045

hendrycksTest-clinical_knowledge acc 0.291 ± 0.028 0.370 ± 0.028

acc_norm 0.287 ± 0.028 0.374 ± 0.028

hendrycksTest-college_biology acc 0.250 ± 0.036 0.250 ± 0.036

acc_norm 0.222 ± 0.035 0.271 ± 0.035

hendrycksTest-college_chemistry acc 0.230 ± 0.042 0.350 ± 0.042

acc_norm 0.250 ± 0.044 0.350 ± 0.044

hendrycksTest-college_computer_science acc 0.280 ± 0.045 0.430 ± 0.045

acc_norm 0.270 ± 0.045 0.390 ± 0.045

hendrycksTest-college_mathematics acc 0.200 ± 0.040 0.370 ± 0.040

acc_norm 0.300 ± 0.046 0.350 ± 0.046

hendrycksTest-college_medicine acc 0.254 ± 0.033 0.312 ± 0.033

acc_norm 0.260 ± 0.033 0.306 ± 0.033

hendrycksTest-college_physics acc 0.225 ± 0.042 0.275 ± 0.042

acc_norm 0.245 ± 0.043 0.284 ± 0.043

hendrycksTest-computer_security acc 0.270 ± 0.045 0.290 ± 0.045

acc_norm 0.330 ± 0.047 0.290 ± 0.047

hendrycksTest-conceptual_physics acc 0.247 ± 0.028 0.315 ± 0.028

acc_norm 0.187 ± 0.026 0.319 ± 0.026

hendrycksTest-econometrics acc 0.193 ± 0.037 0.272 ± 0.037

acc_norm 0.228 ± 0.039 0.281 ± 0.039

hendrycksTest-electrical_engineering acc 0.331 ± 0.039 0.386 ± 0.039

acc_norm 0.338 ± 0.039 0.386 ± 0.039

hendrycksTest-elementary_mathematics acc 0.230 ± 0.022 0.280 ± 0.022

acc_norm 0.270 ± 0.023 0.278 ± 0.023

hendrycksTest-formal_logic acc 0.333 ± 0.042 0.310 ± 0.042

acc_norm 0.302 ± 0.041 0.278 ± 0.041

hendrycksTest-global_facts acc 0.240 ± 0.043 0.250 ± 0.043

acc_norm 0.240 ± 0.043 0.260 ± 0.043

hendrycksTest-high_school_biology acc 0.219 ± 0.024 0.335 ± 0.024

acc_norm 0.284 ± 0.026 0.329 ± 0.026

hendrycksTest-high_school_chemistry acc 0.167 ± 0.026 0.207 ± 0.026

acc_norm 0.256 ± 0.031 0.212 ± 0.031

hendrycksTest-high_school_computer_science acc 0.220 ± 0.042 0.290 ± 0.042

acc_norm 0.280 ± 0.045 0.280 ± 0.045

hendrycksTest-high_school_european_history acc 0.267 ± 0.035 0.358 ± 0.035

acc_norm 0.285 ± 0.035 0.358 ± 0.035

hendrycksTest-high_school_geography acc 0.227 ± 0.030 0.359 ± 0.030

acc_norm 0.298 ± 0.033 0.333 ± 0.033

hendrycksTest-high_school_government_and_politics acc 0.207 ± 0.029 0.301 ± 0.029

acc_norm 0.259 ± 0.032 0.311 ± 0.032

hendrycksTest-high_school_macroeconomics acc 0.262 ± 0.022 0.267 ± 0.022

acc_norm 0.267 ± 0.022 0.262 ± 0.022

hendrycksTest-high_school_mathematics acc 0.174 ± 0.023 0.248 ± 0.023

acc_norm 0.244 ± 0.026 0.270 ± 0.026

hendrycksTest-high_school_microeconomics acc 0.256 ± 0.028 0.265 ± 0.028

acc_norm 0.328 ± 0.030 0.277 ± 0.030

hendrycksTest-high_school_physics acc 0.225 ± 0.034 0.212 ± 0.034

acc_norm 0.219 ± 0.034 0.225 ± 0.034

hendrycksTest-high_school_psychology acc 0.253 ± 0.019 0.338 ± 0.019

acc_norm 0.261 ± 0.019 0.330 ± 0.019

hendrycksTest-high_school_statistics acc 0.264 ± 0.030 0.278 ± 0.030

acc_norm 0.338 ± 0.032 0.273 ± 0.032

hendrycksTest-high_school_us_history acc 0.235 ± 0.030 0.230 ± 0.030

acc_norm 0.270 ± 0.031 0.235 ± 0.031

hendrycksTest-high_school_world_history acc 0.270 ± 0.029 0.388 ± 0.029

acc_norm 0.300 ± 0.030 0.392 ± 0.030

hendrycksTest-human_aging acc 0.296 ± 0.031 0.318 ± 0.031

acc_norm 0.238 ± 0.029 0.314 ± 0.029

hendrycksTest-human_sexuality acc 0.336 ± 0.041 0.290 ± 0.041

acc_norm 0.290 ± 0.040 0.290 ± 0.040

hendrycksTest-international_law acc 0.248 ± 0.039 0.322 ± 0.039

acc_norm 0.496 ± 0.046 0.347 ± 0.046

hendrycksTest-jurisprudence acc 0.250 ± 0.042 0.269 ± 0.042

acc_norm 0.426 ± 0.048 0.296 ± 0.048

hendrycksTest-logical_fallacies acc 0.209 ± 0.032 0.258 ± 0.032

acc_norm 0.288 ± 0.036 0.264 ± 0.036

hendrycksTest-machine_learning acc 0.295 ± 0.043 0.250 ± 0.043

acc_norm 0.259 ± 0.042 0.259 ± 0.042

hendrycksTest-management acc 0.184 ± 0.038 0.311 ± 0.038

acc_norm 0.282 ± 0.045 0.330 ± 0.045

hendrycksTest-marketing acc 0.316 ± 0.030 0.432 ± 0.030

acc_norm 0.338 ± 0.031 0.440 ± 0.031

hendrycksTest-medical_genetics acc 0.300 ± 0.046 0.240 ± 0.046

acc_norm 0.370 ± 0.049 0.270 ± 0.049

hendrycksTest-miscellaneous acc 0.281 ± 0.016 0.323 ± 0.016

acc_norm 0.271 ± 0.016 0.328 ± 0.016

hendrycksTest-moral_disputes acc 0.286 ± 0.024 0.350 ± 0.024

acc_norm 0.355 ± 0.026 0.364 ± 0.026

hendrycksTest-moral_scenarios acc 0.234 ± 0.014 0.264 ± 0.014

acc_norm 0.273 ± 0.015 0.269 ± 0.015

hendrycksTest-nutrition acc 0.275 ± 0.026 0.307 ± 0.026

acc_norm 0.359 ± 0.027 0.333 ± 0.027

hendrycksTest-philosophy acc 0.270 ± 0.025 0.305 ± 0.025

acc_norm 0.315 ± 0.026 0.322 ± 0.026

hendrycksTest-prehistory acc 0.256 ± 0.024 0.361 ± 0.024

acc_norm 0.216 ± 0.023 0.364 ± 0.023

hendrycksTest-professional_accounting acc 0.248 ± 0.026 0.230 ± 0.026

acc_norm 0.259 ± 0.026 0.220 ± 0.026

hendrycksTest-professional_law acc 0.267 ± 0.011 0.275 ± 0.011

acc_norm 0.300 ± 0.012 0.284 ± 0.012

hendrycksTest-professional_medicine acc 0.246 ± 0.026 0.290 ± 0.026

acc_norm 0.232 ± 0.026 0.298 ± 0.026

hendrycksTest-professional_psychology acc 0.258 ± 0.018 0.299 ± 0.018

acc_norm 0.253 ± 0.018 0.315 ± 0.018

hendrycksTest-public_relations acc 0.300 ± 0.044 0.364 ± 0.044

acc_norm 0.164 ± 0.035 0.373 ± 0.035

hendrycksTest-security_studies acc 0.339 ± 0.030 0.343 ± 0.030

acc_norm 0.286 ± 0.029 0.286 ± 0.029

hendrycksTest-sociology acc 0.269 ± 0.031 0.403 ± 0.031

acc_norm 0.264 ± 0.031 0.423 ± 0.031

hendrycksTest-us_foreign_policy acc 0.330 ± 0.047 0.390 ± 0.047

acc_norm 0.350 ± 0.048 0.390 ± 0.048

hendrycksTest-virology acc 0.313 ± 0.036 0.325 ± 0.036

acc_norm 0.331 ± 0.037 0.343 ± 0.037

hendrycksTest-world_religions acc 0.304 ± 0.035 0.316 ± 0.035

acc_norm 0.386 ± 0.037 0.339 ± 0.037

logiqa acc 0.201 ± 0.016 0.280 ± 0.016

acc_norm 0.281 ± 0.018 0.283 ± 0.018

mathqa acc 0.247 ± 0.008 0.248 ± 0.008

acc_norm 0.246 ± 0.008 0.239 ± 0.008

mnli acc 0.339 ± 0.005 0.729 ± 0.005

mnli_mismatched acc 0.338 ± 0.005 0.742 ± 0.005

mrpc acc 0.684 ± 0.023 0.701 ± 0.023

f1 0.812 ± 0.016 0.820 ± 0.016

multirc acc 0.016 ± 0.004 0.004 ± 0.004

openbookqa acc 0.234 ± 0.019 0.248 ± 0.019

acc_norm 0.332 ± 0.021 0.318 ± 0.021

piqa acc 0.721 ± 0.010 0.713 ± 0.010

acc_norm 0.729 ± 0.010 0.708 ± 0.010

qnli acc 0.509 ± 0.007 0.761 ± 0.007

qqp acc 0.368 ± 0.002 0.843 ± 0.002

f1 0.538 ± 0.003 0.789 ± 0.003

race acc 0.353 ± 0.015 0.362 ± 0.015

record f1 0.845 ± 0.004 0.779 ± 0.004

em 0.838 ± 0.004 0.770 ± 0.004

rte acc 0.520 ± 0.030 0.729 ± 0.030

sciq acc 0.893 ± 0.010 0.919 ± 0.010

acc_norm 0.828 ± 0.012 0.913 ± 0.012

sst acc 0.789 ± 0.014 0.862 ± 0.014

webqs acc 0.016 ± 0.003 0.071 ± 0.003

wic acc 0.500 ± 0.020 0.517 ± 0.020

winogrande acc 0.575 ± 0.014 0.570 ± 0.014

wnli acc 0.310 ± 0.055 0.563 ± 0.055

wsc acc 0.365 ± 0.047 0.365 ± 0.047

lambada ppl 5.626 ± 0.139 27.796 ± 0.139

acc 0.622 ± 0.007 0.387 ± 0.007

pubmedqa acc 0.565 ± 0.016 0.496 ± 0.016

coqa f1 0.604 ± 0.018 0.598 ± 0.018

em 0.479 ± 0.020 0.480 ± 0.020

drop em 0.026 ± 0.002 0.001 ± 0.002

f1 0.083 ± 0.002 0.033 ± 0.002

math_algebra acc 0.008 ± 0.003 0.025 ± 0.003

math_geometry acc 0.002 ± 0.002 0.021 ± 0.002

math_intermediate_algebra acc 0.004 ± 0.002 0.025 ± 0.002

math_num_theory acc 0.019 ± 0.006 0.046 ± 0.006

math_prealgebra acc 0.001 ± 0.001 0.039 ± 0.001

math_precalc acc 0.005 ± 0.003 0.016 ± 0.003

One shot

Task Metric 2.7B Tuned

anli_r1 acc 0.331 ± 0.015 0.443 ± 0.015

anli_r2 acc 0.307 ± 0.015 0.373 ± 0.015

anli_r3 acc 0.343 ± 0.014 0.423 ± 0.014

arc_challenge acc 0.302 ± 0.013 0.292 ± 0.013

acc_norm 0.323 ± 0.014 0.323 ± 0.014

arc_easy acc 0.634 ± 0.010 0.567 ± 0.010

acc_norm 0.622 ± 0.010 0.562 ± 0.010

boolq acc 0.536 ± 0.009 0.620 ± 0.009

cb acc 0.429 ± 0.067 0.411 ± 0.067

cola mcc 0.001 ± 0.031 0.022 ± 0.031

copa acc 0.770 ± 0.042 0.780 ± 0.042

ethics_cm acc 0.508 ± 0.008 0.625 ± 0.008

ethics_deontology acc 0.511 ± 0.008 0.683 ± 0.008

ethics_justice acc 0.515 ± 0.010 0.604 ± 0.010

ethics_utilitarianism acc 0.490 ± 0.007 0.536 ± 0.007

ethics_virtue acc 0.726 ± 0.006 0.805 ± 0.006

headqa acc 0.230 ± 0.008 0.228 ± 0.008

acc_norm 0.270 ± 0.008 0.275 ± 0.008

hellaswag acc 0.428 ± 0.005 0.386 ± 0.005

acc_norm 0.557 ± 0.005 0.494 ± 0.005

hendrycksTest-abstract_algebra acc 0.220 ± 0.042 0.270 ± 0.042

acc_norm 0.290 ± 0.046 0.260 ± 0.046

hendrycksTest-anatomy acc 0.289 ± 0.039 0.304 ± 0.039

acc_norm 0.230 ± 0.036 0.289 ± 0.036

hendrycksTest-astronomy acc 0.204 ± 0.033 0.322 ± 0.033

acc_norm 0.303 ± 0.037 0.322 ± 0.037

hendrycksTest-business_ethics acc 0.290 ± 0.046 0.320 ± 0.046

acc_norm 0.280 ± 0.045 0.280 ± 0.045

hendrycksTest-clinical_knowledge acc 0.287 ± 0.028 0.351 ± 0.028

acc_norm 0.328 ± 0.029 0.358 ± 0.029

hendrycksTest-college_biology acc 0.215 ± 0.034 0.271 ± 0.034

acc_norm 0.194 ± 0.033 0.271 ± 0.033

hendrycksTest-college_chemistry acc 0.300 ± 0.046 0.330 ± 0.046

acc_norm 0.340 ± 0.048 0.320 ± 0.048

hendrycksTest-college_computer_science acc 0.330 ± 0.047 0.390 ± 0.047

acc_norm 0.310 ± 0.046 0.360 ± 0.046

hendrycksTest-college_mathematics acc 0.200 ± 0.040 0.280 ± 0.040

acc_norm 0.220 ± 0.042 0.270 ± 0.042

hendrycksTest-college_medicine acc 0.254 ± 0.033 0.295 ± 0.033

acc_norm 0.260 ± 0.033 0.283 ± 0.033

hendrycksTest-college_physics acc 0.304 ± 0.046 0.284 ± 0.046

acc_norm 0.333 ± 0.047 0.304 ± 0.047

hendrycksTest-computer_security acc 0.320 ± 0.047 0.270 ± 0.047

acc_norm 0.320 ± 0.047 0.290 ± 0.047

hendrycksTest-conceptual_physics acc 0.268 ± 0.029 0.349 ± 0.029

acc_norm 0.255 ± 0.029 0.345 ± 0.029

hendrycksTest-econometrics acc 0.298 ± 0.043 0.272 ± 0.043

acc_norm 0.298 ± 0.043 0.263 ± 0.043

hendrycksTest-electrical_engineering acc 0.338 ± 0.039 0.324 ± 0.039

acc_norm 0.290 ± 0.038 0.303 ± 0.038

hendrycksTest-elementary_mathematics acc 0.262 ± 0.023 0.275 ± 0.023

acc_norm 0.294 ± 0.023 0.275 ± 0.023

hendrycksTest-formal_logic acc 0.310 ± 0.041 0.310 ± 0.041

acc_norm 0.294 ± 0.041 0.270 ± 0.041

hendrycksTest-global_facts acc 0.200 ± 0.040 0.290 ± 0.040

acc_norm 0.210 ± 0.041 0.290 ± 0.041

hendrycksTest-high_school_biology acc 0.265 ± 0.025 0.342 ± 0.025

acc_norm 0.287 ± 0.026 0.342 ± 0.026

hendrycksTest-high_school_chemistry acc 0.251 ± 0.031 0.232 ± 0.031

acc_norm 0.291 ± 0.032 0.227 ± 0.032

hendrycksTest-high_school_computer_science acc 0.260 ± 0.044 0.280 ± 0.044

acc_norm 0.300 ± 0.046 0.260 ± 0.046

hendrycksTest-high_school_european_history acc 0.267 ± 0.035 0.309 ± 0.035

acc_norm 0.315 ± 0.036 0.321 ± 0.036

hendrycksTest-high_school_geography acc 0.227 ± 0.030 0.348 ± 0.030

acc_norm 0.278 ± 0.032 0.354 ± 0.032

hendrycksTest-high_school_government_and_politics acc 0.290 ± 0.033 0.332 ± 0.033

acc_norm 0.290 ± 0.033 0.321 ± 0.033

hendrycksTest-high_school_macroeconomics acc 0.279 ± 0.023 0.305 ± 0.023

acc_norm 0.267 ± 0.022 0.285 ± 0.022

hendrycksTest-high_school_mathematics acc 0.252 ± 0.026 0.278 ± 0.026

acc_norm 0.296 ± 0.028 0.304 ± 0.028

hendrycksTest-high_school_microeconomics acc 0.265 ± 0.029 0.256 ± 0.029

acc_norm 0.324 ± 0.030 0.273 ± 0.030

hendrycksTest-high_school_physics acc 0.205 ± 0.033 0.205 ± 0.033

acc_norm 0.232 ± 0.034 0.212 ± 0.034

hendrycksTest-high_school_psychology acc 0.251 ± 0.019 0.328 ± 0.019

acc_norm 0.270 ± 0.019 0.325 ± 0.019

hendrycksTest-high_school_statistics acc 0.319 ± 0.032 0.241 ± 0.032

acc_norm 0.319 ± 0.032 0.245 ± 0.032

hendrycksTest-high_school_us_history acc 0.265 ± 0.031 0.221 ± 0.031

acc_norm 0.260 ± 0.031 0.230 ± 0.031

hendrycksTest-high_school_world_history acc 0.283 ± 0.029 0.371 ± 0.029

acc_norm 0.266 ± 0.029 0.380 ± 0.029

hendrycksTest-human_aging acc 0.296 ± 0.031 0.296 ± 0.031

acc_norm 0.274 ± 0.030 0.291 ± 0.030

hendrycksTest-human_sexuality acc 0.351 ± 0.042 0.290 ± 0.042

acc_norm 0.282 ± 0.039 0.290 ± 0.039

hendrycksTest-international_law acc 0.248 ± 0.039 0.322 ± 0.039

acc_norm 0.347 ± 0.043 0.331 ± 0.043

hendrycksTest-jurisprudence acc 0.269 ± 0.043 0.296 ± 0.043

acc_norm 0.370 ± 0.047 0.296 ± 0.047

hendrycksTest-logical_fallacies acc 0.202 ± 0.032 0.276 ± 0.032

acc_norm 0.270 ± 0.035 0.258 ± 0.035

hendrycksTest-machine_learning acc 0.295 ± 0.043 0.250 ± 0.043

acc_norm 0.330 ± 0.045 0.223 ± 0.045

hendrycksTest-management acc 0.282 ± 0.045 0.320 ± 0.045

acc_norm 0.272 ± 0.044 0.350 ± 0.044

hendrycksTest-marketing acc 0.303 ± 0.030 0.415 ± 0.030

acc_norm 0.329 ± 0.031 0.423 ± 0.031

hendrycksTest-medical_genetics acc 0.330 ± 0.047 0.300 ± 0.047

acc_norm 0.420 ± 0.050 0.300 ± 0.050

hendrycksTest-miscellaneous acc 0.319 ± 0.017 0.318 ± 0.017

acc_norm 0.319 ± 0.017 0.313 ± 0.017

hendrycksTest-moral_disputes acc 0.298 ± 0.025 0.341 ± 0.025

acc_norm 0.318 ± 0.025 0.344 ± 0.025

hendrycksTest-moral_scenarios acc 0.267 ± 0.015 0.240 ± 0.015

acc_norm 0.265 ± 0.015 0.238 ± 0.015

hendrycksTest-nutrition acc 0.278 ± 0.026 0.330 ± 0.026

acc_norm 0.337 ± 0.027 0.350 ± 0.027

hendrycksTest-philosophy acc 0.251 ± 0.025 0.315 ± 0.025

acc_norm 0.293 ± 0.026 0.325 ± 0.026

hendrycksTest-prehistory acc 0.244 ± 0.024 0.352 ± 0.024

acc_norm 0.250 ± 0.024 0.361 ± 0.024

hendrycksTest-professional_accounting acc 0.287 ± 0.027 0.213 ± 0.027

acc_norm 0.248 ± 0.026 0.216 ± 0.026

hendrycksTest-professional_law acc 0.273 ± 0.011 0.267 ± 0.011

acc_norm 0.269 ± 0.011 0.269 ± 0.011

hendrycksTest-professional_medicine acc 0.301 ± 0.028 0.301 ± 0.028

acc_norm 0.268 ± 0.027 0.327 ± 0.027

hendrycksTest-professional_psychology acc 0.279 ± 0.018 0.304 ± 0.018

acc_norm 0.284 ± 0.018 0.310 ± 0.018

hendrycksTest-public_relations acc 0.327 ± 0.045 0.345 ± 0.045

acc_norm 0.309 ± 0.044 0.336 ± 0.044

hendrycksTest-security_studies acc 0.265 ± 0.028 0.331 ± 0.028

acc_norm 0.208 ± 0.026 0.290 ± 0.026

hendrycksTest-sociology acc 0.269 ± 0.031 0.393 ± 0.031

acc_norm 0.249 ± 0.031 0.383 ± 0.031

hendrycksTest-us_foreign_policy acc 0.290 ± 0.046 0.320 ± 0.046

acc_norm 0.320 ± 0.047 0.320 ± 0.047

hendrycksTest-virology acc 0.289 ± 0.035 0.349 ± 0.035

acc_norm 0.265 ± 0.034 0.355 ± 0.034

hendrycksTest-world_religions acc 0.374 ± 0.037 0.345 ± 0.037

acc_norm 0.409 ± 0.038 0.351 ± 0.038

logiqa acc 0.255 ± 0.017 0.273 ± 0.017

acc_norm 0.272 ± 0.017 0.280 ± 0.017

mathqa acc 0.256 ± 0.008 0.253 ± 0.008

acc_norm 0.258 ± 0.008 0.240 ± 0.008

mnli acc 0.338 ± 0.005 0.801 ± 0.005

mnli_mismatched acc 0.362 ± 0.005 0.811 ± 0.005

mrpc acc 0.571 ± 0.025 0.750 ± 0.025

f1 0.689 ± 0.022 0.841 ± 0.022

multirc acc 0.047 ± 0.007 0.012 ± 0.007

openbookqa acc 0.222 ± 0.019 0.268 ± 0.019

acc_norm 0.346 ± 0.021 0.344 ± 0.021

piqa acc 0.726 ± 0.010 0.714 ± 0.010

acc_norm 0.736 ± 0.010 0.718 ± 0.010

qnli acc 0.504 ± 0.007 0.788 ± 0.007

qqp acc 0.534 ± 0.002 0.847 ± 0.002

f1 0.372 ± 0.004 0.793 ± 0.004

race acc 0.352 ± 0.015 0.355 ± 0.015

record f1 0.843 ± 0.004 0.778 ± 0.004

em 0.835 ± 0.004 0.771 ± 0.004

rte acc 0.491 ± 0.030 0.747 ± 0.030

sciq acc 0.930 ± 0.008 0.939 ± 0.008

acc_norm 0.938 ± 0.008 0.935 ± 0.008

sst acc 0.492 ± 0.017 0.916 ± 0.017

webqs acc 0.054 ± 0.005 0.095 ± 0.005

wic acc 0.472 ± 0.020 0.539 ± 0.020

winogrande acc 0.582 ± 0.014 0.571 ± 0.014

wnli acc 0.380 ± 0.058 0.549 ± 0.058

wsc acc 0.365 ± 0.047 0.365 ± 0.047

lambada ppl 6.423 ± 0.162 20.150 ± 0.162

acc 0.576 ± 0.007 0.394 ± 0.007

pubmedqa acc 0.529 ± 0.016 0.479 ± 0.016

coqa f1 0.606 ± 0.018 0.581 ± 0.018

em 0.484 ± 0.020 0.472 ± 0.020

drop em 0.001 ± 0.000 0.001 ± 0.000

f1 0.039 ± 0.001 0.031 ± 0.001

math_algebra acc 0.016 ± 0.004 0.024 ± 0.004

math_counting_and_prob acc 0.023 ± 0.007 0.030 ± 0.007

math_geometry acc 0.006 ± 0.004 0.021 ± 0.004

math_intermediate_algebra acc 0.020 ± 0.005 0.029 ± 0.005

math_num_theory acc 0.037 ± 0.008 0.039 ± 0.008

math_prealgebra acc 0.023 ± 0.005 0.041 ± 0.005

math_precalc acc 0.015 ± 0.005 0.022 ± 0.005

The model can be downloaded here, though I don't recommend using it for anything.