tutorials/intro/Intro_Tutorial_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Training an End Extraction Model\n",
    "\n",
    "In this final section of the tutorial, we'll use the noisy training labels we generated in the last tutorial part to train our end extraction model.\n",
    "\n",
    "For this tutorial, we will be training a Bi-LSTM, a state-of-the-art deep neural network implemented in [TensorFlow](https://www.tensorflow.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reload the probabilistic training labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_marginals\n",
    "\n",
    "train_marginals = load_marginals(session, split=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also reload the candidates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_cands = session.query(Spouse).filter(Spouse.split == 0).order_by(Spouse.id).all()\n",
    "dev_cands   = session.query(Spouse).filter(Spouse.split == 1).order_by(Spouse.id).all()\n",
    "test_cands  = session.query(Spouse).filter(Spouse.split == 2).order_by(Spouse.id).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we load gold labels for evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev  = load_gold_labels(session, annotator_name='gold', split=1)\n",
    "L_gold_test = load_gold_labels(session, annotator_name='gold', split=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can setup our discriminative model. Here we specify the model and learning hyperparameters.\n",
    "\n",
    "They can also be set automatically using a search based on the dev set with a [GridSearch](https://github.com/HazyResearch/snorkel/blob/master/snorkel/learning/utils.py) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[reRNN] Training model\n",
      "[reRNN] n_train=17211  #epochs=10  batch size=256\n",
      "[reRNN] Epoch 0 (23.14s)\tAverage loss=0.574778\tDev F1=24.47\n",
      "[reRNN] Epoch 1 (50.09s)\tAverage loss=0.540465\tDev F1=35.18\n",
      "[reRNN] Epoch 2 (76.51s)\tAverage loss=0.540333\tDev F1=19.47\n",
      "[reRNN] Epoch 3 (98.49s)\tAverage loss=0.540295\tDev F1=37.00\n",
      "[reRNN] Epoch 4 (123.53s)\tAverage loss=0.538588\tDev F1=34.94\n",
      "[reRNN] Epoch 5 (148.63s)\tAverage loss=0.537206\tDev F1=37.15\n",
      "[reRNN] Epoch 6 (170.56s)\tAverage loss=0.536529\tDev F1=40.92\n",
      "[reRNN] Epoch 7 (193.84s)\tAverage loss=0.536493\tDev F1=40.78\n",
      "[reRNN] Epoch 8 (218.16s)\tAverage loss=0.536371\tDev F1=41.22\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 9 (244.28s)\tAverage loss=0.536352\tDev F1=40.08\n",
      "[reRNN] Training done (245.50s)\n",
      "INFO:tensorflow:Restoring parameters from checkpoints/reRNN/reRNN-8\n",
      "[reRNN] Loaded model <reRNN>\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning.disc_models.rnn import reRNN\n",
    "\n",
    "train_kwargs = {\n",
    "    'lr':         0.01,\n",
    "    'dim':        50,\n",
    "    'n_epochs':   10,\n",
    "    'dropout':    0.25,\n",
    "    'print_freq': 1,\n",
    "    'max_sentence_length': 100\n",
    "}\n",
    "\n",
    "lstm = reRNN(seed=1701, n_threads=None)\n",
    "lstm.train(train_cands, train_marginals, X_dev=dev_cands, Y_dev=L_gold_dev, **train_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we get the precision, recall, and F1 score from the discriminative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prec: 0.385, Recall: 0.601, F1 Score: 0.470\n"
     ]
    }
   ],
   "source": [
    "p, r, f1 = lstm.score(test_cands, L_gold_test)\n",
    "print(\"Prec: {0:.3f}, Recall: {1:.3f}, F1 Score: {2:.3f}\".format(p, r, f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the candidates returned in sets (true positives, false positives, true negatives, false negatives) as well as a more detailed score report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 0.601\n",
      "Neg. class accuracy: 0.916\n",
      "Precision            0.385\n",
      "Recall               0.601\n",
      "F1                   0.47\n",
      "----------------------------------------\n",
      "TP: 131 | FP: 209 | TN: 2274 | FN: 87\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = lstm.error_analysis(session, test_cands, L_gold_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if this is the final test set that you will be reporting final numbers on, to avoid biasing results you should not inspect results.  However you can run the model on your _development set_ and, as we did in the previous part with the generative labeling function model, inspect examples to do error analysis.\n",
    "\n",
    "You can also improve performance substantially by increasing the number of training epochs!\n",
    "\n",
    "Finally, we can save the predictions of the model on the test set back to the database. (This also works for other candidate sets, such as unlabeled candidates.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 2701 marginals\n"
     ]
    }
   ],
   "source": [
    "lstm.save_marginals(session, test_cands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### More importantly, you completed the introduction to Snorkel! Give yourself a pat on the back!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2.7 (snorkel-0.6)",
   "language": "python",
   "name": "snorkel-0.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}