Skip to content

Author implementation of the paper "CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge"

Notifications You must be signed in to change notification settings

jonathanherzig/commonsenseqa

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 

Repository files navigation

CommonsenseQA

A question answering dataset for commonsense reasoning.

Check out the website!

Downloading the Data

You can download the data from the website, which also has an evaluation script. The leaderboard is for the random split of the data.

Running ESIM

Our implementation is based on this code. To run it, follow these steps:

  1. Install ESIM dependencies:
    cd esim
    pip install -r requirements.txt
    cd ..
    
  2. Place the dataset in data/ folder.
  3. Set PYTHONPATH to the commonsenseqa directory: export PYTHONPATH=$(pwd)
  4. Run the model either with pre-trained GloVe embeddings:
    python -m allennlp.run train esim/train-glove-csqa.json -s tmp --include-package esim
    
  5. Alternatively, run the model with ELMo pretrained contextual embeddings:
    python -m allennlp.run train esim/train-elmo-csqa.json -s tmp --include-package esim
    

Running BERT

To run BERT on CommonsenseQA, first install the BERT dependencies:

cd bert/
pip install -r requirements.txt

Then, obtain the CommonsenseQA data, and download the pretrained BERT weights. For the paper, we used BERT Large, Uncased. To train BERT Large, you'll most likely need to use a TPU; BERT base can be trained on a standard GPU.

To run training:

GPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

TPU

python run_commonsense_qa.py
  --split=$SPLIT \
  --use_tpu=true \
  --tpu_name=$TPU_NAME \
  --do_train=true \
  --do_eval=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$BERT_DIR/bert_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=16 \
  --learning_rate=2e-5 \
  --num_train_epochs=3.0 \
  --output_dir=$OUTPUT_DIR

For TPUs, all directories must be in Google Storage. The environment variables have the following meanings:

  • $SPLIT should either be rand or qtoken, depending on the split you'd like to run.
  • $DATA_DIR is a location for the CommonsenseQA data.
  • $BERT_DIR is a location for the pre-trained BERT files.
  • $TPU_NAME is the name of the TPU.
  • $OUTPUT_DIR is the directory to write output to.

To predict on the test set, run:

GPU (only)

python run_commonsense_qa.py \
  --split=$SPLIT \
  --do_predict=true \
  --data_dir=$DATA_DIR \
  --vocab_file=$BERT_DIR/vocab.txt \
  --bert_config_file=$BERT_DIR/bert_config.json \
  --init_checkpoint=$TRAINED_CHECKPOINT \
  --max_seq_length=128 \
  --output_dir=$OUTPUT_DIR

Prediction must be run on a GPU (including for BERT Large). All environment variables have the same meanings, and the new variable $TRAINED_CHECKPOINT is simply the prefix for your trained checkpoint files from fine-tuning BERT. It should look something like $OUTPUT_DIR/model.ckpt-1830.

About

Author implementation of the paper "CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages