In this advanced tutorial, we will build an Emmental application to tackle the SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks.
To use this tutorial you will need to:
- Install python dependencies for this tutorial.
pip install -r requirements.txt- Download the SuperGLUE and other datasets (e.g., SWAG) used in the tutorial to the local directory.
- Run the SuperGLUE task with the following command, where TASK is one of cb, copa, multirc, rte, wic, wsc and external task for pretraining (e.g., swag).
bash run_superglue.sh [TASK] [DATA] [SEED] [GPU_ID]We use pretrained models to improve model perfromance, for example we use MNLI model from GLUE tutorial for RTE and CB task, and use SWAG model to improve COPA task. To get the pretrained model, you can train MNLI model using GLUE tutorial and SWAG using this tutorial.
Note: Due to small validation sets and evidence of overfitting to the validation set on some of the tasks (CB, COPA, and WiC), we also recommend using cross-validation on these tasks. For each, we ran k-fold cross-validation with a value of k that would result in a validation set of approximately the same size as the original provided split.
Much of the code in this tutorial was adapted from the jiant. We thank all authors to provide these available online.