This is companion code for our NAACL 2021 paper, "How many data points is a prompt worth?". Check out our interactive blog post and paper for more details.
You should place your MNLI/SuperGLUE data in data in folders named after each task. For SuperGLUE data, run python superglue_data_splitting first in order to have a dev/test split.
You may run a prompted experiment with .scripts/sequential_pvps.sh and a head experiment with .scripts/sequential_supervised.sh. Modify those scripts to change the SuperGLUE task you wish to train on.
Originally forked from https://github.com/timoschick/pet.