LP2-Lecture11-dynet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Language Processing 2\n",
    "\n",
    "### Session 11\n",
    "\n",
    "#### Jurgen Wedekind & Manex Agirrezabal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Good sources of information:\n",
    "\n",
    " - https://dynet.readthedocs.io/en/latest/python_ref.html\n",
    " - https://dynet.readthedocs.io/en/latest/tutorials_notebooks/RNNs.html\n",
    " \n",
    " - http://phontron.com/slides/emnlp2016-dynet-tutorial-part1.pdf\n",
    " - http://phontron.com/slides/emnlp2016-dynet-tutorial-part2.pdf\n",
    " \n",
    " - https://github.com/clab/dynet/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:20.211894Z",
     "start_time": "2019-05-03T11:47:19.690132Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import dynet as dy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```\n",
    "conda create -n superdynet python=3.6 anaconda\n",
    "conda activate superdynet\n",
    "pip install dynet\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The goal\n",
    "\n",
    "  - Intro to DyNet\n",
    "  - Gradient Descent (exercise with NLP)\n",
    "  - Recurrent Neural Networks with DyNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Intro to DyNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    " - Computation Graph\n",
    " - Expressions (~nodes in the graph)\n",
    " - Parameters\n",
    " - Model - a collection of parameters (`ParameterCollection`)\n",
    " - Trainer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Computation graphs\n",
    "\n",
    " - Deep Learning’s Lingua Franca\n",
    " - Page 13: http://phontron.com/slides/emnlp2016-dynet-tutorial-part1.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Expressions\n",
    "\n",
    "Expressions are the main data types being manipulated in a DyNet program. Each expression represents a sub-computation in a computation graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameters\n",
    "\n",
    "Parameters are things that are optimized.\n",
    "\n",
    "#### Lookup parameters\n",
    "\n",
    "LookupParameters represents a table of parameters.\n",
    "\n",
    "They are used to embed a set of discrete objects (e.g. word embeddings)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Model or `ParameterCollection`\n",
    "\n",
    "A colllection of parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Trainer\n",
    "\n",
    "This is the one that will update the weights.\n",
    "\n",
    " - Initialize a Trainer with a given model.\n",
    " - Compute gradients by calling `expr.backward()` from a scalar node.\n",
    " - Call `trainer.update()` to update the model parameters using the gradients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training with DyNet (today)\n",
    "\n",
    " - Create model, add parameters, create trainer.\n",
    " - For each training example:\n",
    "    - create computation graph for the loss\n",
    "    - run forward (compute the loss)\n",
    "    - run backward (compute the gradients)\n",
    "    - update parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:21.266388Z",
     "start_time": "2019-05-03T11:47:20.216850Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:21.285352Z",
     "start_time": "2019-05-03T11:47:21.271840Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def linearly_seperable_data(num_instances, epsilon=0.1):\n",
    "    '''\n",
    "    generates a linearly separable data set\n",
    "    '''\n",
    "    # fill the first column (the labels) randomly with -1s and 1s\n",
    "    labels = 2 * np.random.randint(0, 2, num_instances) - 1\n",
    "    # pick x1 at random\n",
    "    x1 = np.random.random(num_instances)\n",
    "    # base x2 off of x1, add random noise and epsilon, and move up or down\n",
    "    x2 = x1 + ((epsilon + np.random.random(num_instances)) * labels)\n",
    "    labels = ((labels+1)/2).astype(int)\n",
    "    return pd.DataFrame({'x1': x1, 'x2': x2, 'label': labels})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:21.340428Z",
     "start_time": "2019-05-03T11:47:21.292179Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x1</th>\n",
       "      <th>x2</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.916664</td>\n",
       "      <td>0.067679</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002023</td>\n",
       "      <td>0.941277</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.971332</td>\n",
       "      <td>0.825757</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.889048</td>\n",
       "      <td>1.494691</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.699489</td>\n",
       "      <td>1.388303</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         x1        x2  label\n",
       "0  0.916664  0.067679      0\n",
       "1  0.002023  0.941277      1\n",
       "2  0.971332  0.825757      0\n",
       "3  0.889048  1.494691      1\n",
       "4  0.699489  1.388303      1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(24)\n",
    "l_s_d = linearly_seperable_data(100)\n",
    "l_s_d.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:21.363605Z",
     "start_time": "2019-05-03T11:47:21.346742Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((100, 2), (100,))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = l_s_d[['x1','x2']].values\n",
    "y = l_s_d['label']\n",
    "X.shape,y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:21.732622Z",
     "start_time": "2019-05-03T11:47:21.366613Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.640983813405037\n",
      "0.5310313957929611\n",
      "0.4631859940290451\n",
      "0.41374624490737916\n",
      "0.3766621547937393\n",
      "0.3481876826286316\n",
      "0.3257425856590271\n",
      "0.30759513318538667\n",
      "0.2925882142782211\n",
      "0.27993600726127627\n",
      "0.269091460108757\n",
      "0.25966390073299406\n",
      "0.2513677132129669\n",
      "0.24398960888385773\n",
      "0.23736720263957978\n",
      "0.2313747549057007\n",
      "0.2259134656190872\n",
      "0.22090457916259765\n",
      "0.21628454566001892\n",
      "0.2120015001296997\n",
      "0.2080126655101776\n",
      "0.20428249180316926\n",
      "0.20078113913536072\n",
      "0.19748337626457213\n",
      "0.19436767160892487\n",
      "0.19141557812690735\n",
      "0.18861126720905305\n",
      "0.18594085693359375\n",
      "0.18339235246181487\n",
      "0.18095520675182342\n",
      "0.1786201375722885\n",
      "0.17637895286083222\n",
      "0.17422431349754333\n",
      "0.17214976966381074\n",
      "0.17014945447444915\n",
      "0.16821819841861724\n",
      "0.1663512843847275\n",
      "0.16454450011253358\n",
      "0.16279396891593934\n",
      "0.16109627544879912\n",
      "0.15944820404052734\n",
      "0.1578469008207321\n",
      "0.15628962457180023\n",
      "0.15477409243583679\n",
      "0.15329797327518463\n",
      "0.15185929834842682\n",
      "0.15045614063739776\n",
      "0.14908674359321594\n",
      "0.14774954438209534\n",
      "0.14644303381443025\n",
      "0.14516584157943727\n",
      "0.14391663610935213\n",
      "0.14269424438476563\n",
      "0.14149756014347076\n",
      "0.1403255444765091\n",
      "0.1391772049665451\n",
      "0.1380516231060028\n",
      "0.13694795489311218\n",
      "0.13586537837982177\n",
      "0.1348031359910965\n",
      "0.13376059412956237\n",
      "0.13273695826530457\n",
      "0.13173167884349823\n",
      "0.13074418663978576\n",
      "0.1297738629579544\n",
      "0.12882018804550172\n",
      "0.12788264334201813\n",
      "0.1269607150554657\n",
      "0.12605403006076812\n",
      "0.12516207933425905\n",
      "0.12428448677062988\n",
      "0.12342081665992737\n",
      "0.12257075011730194\n",
      "0.12173388600349426\n",
      "0.12090990662574769\n",
      "0.12009852290153504\n",
      "0.11929929375648499\n",
      "0.11851199805736541\n",
      "0.11773635506629944\n",
      "0.11697209417819977\n",
      "0.11621892511844635\n",
      "0.11547659873962403\n",
      "0.11474488019943237\n",
      "0.11402348041534424\n",
      "0.11331218719482422\n",
      "0.11261080503463745\n",
      "0.11191914319992065\n",
      "0.11123693108558655\n",
      "0.11056400001049042\n",
      "0.10990015804767608\n",
      "0.10924522757530213\n",
      "0.10859897196292877\n",
      "0.10796125173568726\n",
      "0.10733192682266235\n",
      "0.10671078979969024\n",
      "0.10609767198562622\n",
      "0.10549241423606873\n",
      "0.10489487767219544\n",
      "0.10430495321750641\n",
      "0.10372238099575043\n"
     ]
    }
   ],
   "source": [
    "#MODEL SPECIFICATION\n",
    "model = dy.ParameterCollection()\n",
    "W_p = model.add_parameters((2, 2))\n",
    "b_p = model.add_parameters (2)\n",
    "\n",
    "trainer = dy.SimpleSGDTrainer (model)\n",
    "\n",
    "num_epochs = 100\n",
    "closses = []\n",
    "\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    closs = 0\n",
    "\n",
    "    for input_x, out_label in zip(X,y):\n",
    "        dy.renew_cg()\n",
    "        x = dy.inputVector(input_x) #This must go after renewing the graph\n",
    "        \n",
    "        output_value = dy.logistic(W_p*x+b_p)\n",
    "        \n",
    "        loss = dy.hinge(output_value,out_label)\n",
    "\n",
    "        loss_val = loss.value()\n",
    "        closs = closs + loss_val\n",
    "        \n",
    "        loss.backward ()\n",
    "        trainer.update()\n",
    "    \n",
    "    print (closs/len(l_s_d))\n",
    "    closses.append(closs/len(l_s_d))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:43.850939Z",
     "start_time": "2019-05-03T11:47:43.613279Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAH8FJREFUeJzt3Xt4XHd95/H3V6MZSaP73bZkS7ZjJ3Gcu+NcgIRAsk24OHRhaWhpQylr2KdpaGG7Dds2benutnRbaEuzbNPQFGhpmgYKDg2kkJCGSxJ8S0J8v8qWr7rfpdHl2z9mpEwcyRrbGo3mzOf1PPPMnDM/zXzPc6TPOfqd3znH3B0REQmWvEwXICIic0/hLiISQAp3EZEAUriLiASQwl1EJIAU7iIiAaRwFxEJIIW7iEgAKdxFRAIoP1NfXFNT483NzZn6ehGRrLR169Z2d6+drV3Gwr25uZktW7Zk6utFRLKSmbWk0k7dMiIiAaRwFxEJIIW7iEgAKdxFRAJI4S4iEkAKdxGRAFK4i4gEUNaF++bDnXzmO7uZmNDtAUVEZpJ14f7y0W6+8OwB+kbGMl2KiMiClXXhXhmNANA1EMtwJSIiC1fWhXtVcSLcBxXuIiIzybpwr4iGAYW7iMjZZF24T+25D4xmuBIRkYUr68K9IqpuGRGR2WRduJcV5hPKM4W7iMhZZF24mxmV0TCd6pYREZlR1oU7xLtmurXnLiIyo6wM96poRN0yIiJnkZXhXhENa7SMiMhZZGW4VxVrz11E5GxSCnczu8PM9pjZfjO7f4Y27zeznWa2w8y+Ordlvl5FolvGXRcPExGZTv5sDcwsBDwI3A60ApvNbJO770xqswr4FPAmd+8ys7p0FQxQVRxmdNwZiI1TUjDrIoiI5JxU9tzXA/vd/aC7x4BHgbvOaPNfgQfdvQvA3U/PbZmvV6GLh4mInFUq4d4AHE2abk3MS7YaWG1mPzKzF8zsjuk+yMw2mtkWM9vS1tZ2fhUTHy0DOktVRGQmqYS7TTPvzM7ufGAV8FbgA8DDZlbxhh9yf8jd17n7utra2nOtdUplcfziYZ3acxcRmVYq4d4KLE2abgSOT9Pmm+4+6u6HgD3Ewz4tJrtlugc1HFJEZDqphPtmYJWZLTezCHA3sOmMNt8AbgUwsxri3TQH57LQZOqWERE5u1nD3d3HgHuBp4BdwGPuvsPMPm1mGxLNngI6zGwn8H3gN929I11FlxWFMdMBVRGRmaQ0jtDdnwSePGPeA0mvHfhE4pF2oTyjoihMl7plRESmlZVnqEL8Xqqd6pYREZlW9oZ7sa4MKSIyk+wNd13TXURkRlkc7tpzFxGZSfaGe3FEJzGJiMwge8M9GmFkbIKh2HimSxERWXCyONwTlyBQ14yIyBtkbbjrypAiIjPL2nCvKtb1ZUREZpK14a5uGRGRmWVvuE/tuSvcRUTOlLXhXlGka7qLiMwka8M9P5RHWWG++txFRKaRteEOOpFJRGQm2R3u0Yhu2CEiMo0sD/ewwl1EZBrZHe7FEbp0ZUgRkTfI7nDXlSFFRKaV5eEeZiA2zsiYLh4mIpIsu8O9ePL6MuqaERFJltXhvri8EIDjPUMZrkREZGHJ6nBvrIwC0NqlcBcRSZbV4d5QUQTAMYW7iMjrZHW4FxfkUxkN09o1mOlSREQWlKwOd4CGyiKOdWvPXUQkWdaHe2NFVH3uIiJnyPpwb6gsorVrEHfPdCkiIgtGSuFuZneY2R4z229m90/z/ofMrM3MXko8PjL3pU6vsbKI4dEJXR1SRCRJ/mwNzCwEPAjcDrQCm81sk7vvPKPpP7n7vWmo8awmR8y0dg1RXVIw318vIrIgpbLnvh7Y7+4H3T0GPArcld6yUjc51l0HVUVEXpNKuDcAR5OmWxPzzvReM3vFzB43s6VzUl0KGion99w1HFJEZFIq4W7TzDvz6OUTQLO7XwF8D/jStB9kttHMtpjZlra2tnOrdAblRWFKC/N1IpOISJJUwr0VSN4TbwSOJzdw9w53H0lM/g1w7XQf5O4Pufs6d19XW1t7PvVOq6GiSMMhRUSSpBLum4FVZrbczCLA3cCm5AZmtjhpcgOwa+5KnF1jZVR97iIiSWYdLePuY2Z2L/AUEAL+1t13mNmngS3uvgm4z8w2AGNAJ/ChNNb8Bo2VRbxwsAN3x2y6XiQRkdwya7gDuPuTwJNnzHsg6fWngE/NbWmpa6wson9kjN6hMcqj4UyVISKyYGT9GaoQD3eAoxoxIyICBCTcGyo01l1EJFkgwr2x8rWzVEVEJCDhXhENE42ENNZdRCQhEOFuZjQmrg4pIiIBCXfQiUwiIskCE+46kUlE5DWBCfeGyiJ6hkbpGx7NdCkiIhkXmHCfGuveqb13EZHAhPvK2hIA9rf1Z7gSEZHMC0y4r6gtJpRn7DvVl+lSREQyLjDhXpAfoqk6yp6TCncRkcCEO8DF9aXsO61uGRGRQIX7qvpSWjoGGB4dz3QpIiIZFahwX11fwoTDAR1UFZEcF7BwLwVgrw6qikiOC1S4N1cXEw4Ze09pz11Eclugwj2Sn8fymmINhxSRnBeocIf4QdU9CncRyXGBC/eL60s52jnEYGws06WIiGRM4MJ9dX3iMgQa7y4iOSxw4b5qasSMwl1Eclfgwr2pKkokP0/DIUUkpwUu3PNDeaysLVG4i0hOC1y4Q7zffZ+6ZUQkhwU03Es51j2kuzKJSM4KbLiDDqqKSO4KZLivbSgD4JXW7gxXIiKSGSmFu5ndYWZ7zGy/md1/lnbvMzM3s3VzV+K5W1xexJLyQra2dGWyDBGRjJk13M0sBDwI3AmsAT5gZmumaVcK3Ae8ONdFno9rmirZpnAXkRyVyp77emC/ux909xjwKHDXNO3+EPgTYHgO6ztv1yyr5HjPMCd6hjJdiojIvEsl3BuAo0nTrYl5U8zsamCpu39rDmu7INc2VQKwrUX97iKSe1IJd5tmnk+9aZYHfA745KwfZLbRzLaY2Za2trbUqzwPa5aUURjOU7+7iOSkVMK9FViaNN0IHE+aLgXWAs+a2WHgBmDTdAdV3f0hd1/n7utqa2vPv+oUhEN5XNFQwbYjCncRyT2phPtmYJWZLTezCHA3sGnyTXfvcfcad29292bgBWCDu29JS8Xn4JqmSnYc79ENs0Uk58wa7u4+BtwLPAXsAh5z9x1m9mkz25DuAi/EtU2VjI47Pz3Wk+lSRETmVX4qjdz9SeDJM+Y9MEPbt154WXPj6mUVAGxr6eK65qoMVyMiMn8CeYbqpJqSApqrozqoKiI5J9DhDvHx7tuOdOHuszcWEQmI4Id7UyXt/TGOdupkJhHJHYEP98mTmV441JHhSkRE5k/gw/2SRaXUlRbw7J7TmS5FRGTeBD7czYy3XVLHD/a2Mzo+kelyRETmReDDHeDWS+roGxlj8+HOTJciIjIvciLc33xRDZFQHt/fra4ZEckNORHuxQX5XL+iimcU7iKSI3Ii3AFuvbiOA20DtHQMZLoUEZG0y5lwf9sldQDaexeRnJAz4d5cU8yK2mKFu4jkhJwJd4C3XVzHiwc7GRgZy3QpIiJplVvhfkkdsfEJfri/PdOliIikVU6F+3XLq6gqjrDppeOzNxYRyWI5Fe7hUB4brlzCd3eeomdwNNPliIikTU6FO8D7rm0kNj7BE69o711Egivnwv2yJWVcXF/K41tbM12KiEja5Fy4mxnvvbaBl452c6CtP9PliIikRc6FO8B7rmoglGd8TXvvIhJQORnudWWF3Lyqhq9vO8b4hG6/JyLBk5PhDvDeaxs52TvMjw9ozLuIBE/Ohvttl9ZTGQ3z5edbMl2KiMicy9lwLwyH+MUbm/nuzlPsP60DqyISLDkb7gD33NhEQX4ef/PcwUyXIiIyp3I63KtLCnj/uqX8y/ZjnOodznQ5IiJzJqfDHeAjb1nO2MQEj/zocKZLERGZMzkf7k3Vxdx5+WL+4YUW+oZ1vRkRCYaUwt3M7jCzPWa238zun+b9j5nZT83sJTP7oZmtmftS0+ejN6+gb2SMv3/hSKZLERGZE7OGu5mFgAeBO4E1wAemCe+vuvvl7n4V8CfAZ+e80jS6orGCW1bX8oVn99M1EMt0OSIiFyyVPff1wH53P+juMeBR4K7kBu7emzRZDGTdaZ//8x2X0j8yxuef2Z/pUkRELlgq4d4AHE2abk3Mex0z+1UzO0B8z/2+uSlv/ly8qJSfu24pX3nhMIfbBzJdjojIBUkl3G2aeW/YM3f3B919JfBbwO9M+0FmG81si5ltaWtrO7dK58Fv3L6acCiPz3xnd6ZLERG5IKmEeyuwNGm6ETjbnS4eBd4z3Rvu/pC7r3P3dbW1talXOU/qSgv52C0r+farJ9l8uDPT5YiInLdUwn0zsMrMlptZBLgb2JTcwMxWJU2+E9g3dyXOr4+8ZTmLygr53W+8SmxsItPliIicl1nD3d3HgHuBp4BdwGPuvsPMPm1mGxLN7jWzHWb2EvAJ4J60VZxm0Ug+f/ietew+2cf/e1YHV0UkO+Wn0sjdnwSePGPeA0mvPz7HdWXU7WvqueuqJfzVM/v5mcsWceniskyXJCJyTnL+DNWZ/P67L6MiGuY3H3+Z0XF1z4hIdlG4z6CyOMIf3rWWV4/18v+fPZDpckREzonC/SzuvHwx775yCZ/73l5eONiR6XJERFKmcJ/F//nZtTRXF3PvV7dzWpcFFpEsoXCfRWlhmC988FoGRsa496vb1f8uIllB4Z6CixeV8kf/+XJ+criTP/62zl4VkYUvpaGQAu+5uoHtR7r44g8Psawqyj03NWe6JBGRGSncz8HvvmsNx7qH+P0ndlBfVsgdaxdluiQRkWmpW+Yc5Ify+PwHruHKxgo+/uh2trbo+jMisjAp3M9RUSTEF+9Zx+LyQn75kc280tqd6ZJERN5A4X4eqksK+PuPXE9ZUZgPPvwiLx9VwIvIwqJwP0+NlVEe3XgD5dEwH/zii7ykgBeRBUThfgHiAX8jldEIH3z4RX6wb+HdgEREcpPC/QI1VBTx2EdvpLGyiF9+ZDNf29qa6ZJERBTuc2FReSGPfexGrl9RxSf/+WU+//Q+3LPuHuEiEiAK9zlSVhjmkQ+t52evbuDPvruXe7+6nYGRsUyXJSI5SuE+hyL5eXz2/VfyqTsv4duvnuA9D/6IQ+0DmS5LRHKQwn2OmRkfvWUlX/7w9bT3j7Dh8z9k08tnu5+4iMjcU7inyZtX1fDEr72ZVfUl3PeP2/nkYy/Tr24aEZknCvc0aqyM8thHb+S+t6/iX7a38s6//AE/OaRLFohI+inc0yw/lMcnbl/NoxtvZMKd9//18zzwzVe1Fy8iaaVwnyfrl1fx1K/fzIfftJyvvNDCz3zuOf5tx0kNmRSRtFC4z6NoJJ8H3r2Gxz92E8UFITZ+ZSsf/rvNHNaIGhGZYwr3DLi2qZJ/ve8t/M47L2Xz4S7+0+ee44++vYueodFMlyYiAaFwz5BwKI+PvGUFz3zyFt515WIeeu4gb/2/3+eRHx0iNqb7tIrIhVG4Z1hdWSGfff9VPHHvm1mzpIw/eGInt/7ps/zT5iO6GbeInDeF+wKxtqGcv/+V6/nSh9dTUxLht772U2777L/zz1uOKuRF5JxZpkZrrFu3zrds2ZKR717o3J2nd53mc9/by47jvTRUFLHx5hW8f91SiiKhTJcnIhlkZlvdfd1s7VLaczezO8xsj5ntN7P7p3n/E2a208xeMbOnzazpfIqWODPjtjX1fOvX3swjH7qOReWF/N6mHdz0x0/zZ/+2h9N9w5kuUUQWuFn33M0sBOwFbgdagc3AB9x9Z1KbW4EX3X3QzP4b8FZ3/7mzfa723FPn7vzkUCcP//AQ39t1inBeHu+6cjH33NjMlUsrMl2eiMyjVPfc81P4rPXAfnc/mPjgR4G7gKlwd/fvJ7V/AfjguZUrZ2NmXL+imutXVHOofYBHfnSIr21t5evbjnFlYzm/cEMT77piMdFIKqtTRHJBKnvu7wPucPePJKZ/Ebje3e+dof1fASfd/X9N895GYCPAsmXLrm1pabnA8nNX3/AoX992jC8/f5gDbQOUFuSz4aol3H3dMtY2lGFmmS5RRNJgLvfcp0uJabcIZvZBYB1wy3Tvu/tDwEMQ75ZJ4btlBqWFYe65qZlfurGJzYe7ePQnR3h8ayv/8OIRVteX8N5rGvnZqxuoKyvMdKkikgGphHsrsDRpuhF4wwXKzew24LeBW9x9ZG7Kk9mYGeuXV7F+eRW/9+7LeOKV43xtWyt/9O3dfOY7u7lpZQ0brlrCHWsXUVYYznS5IjJPUumWySd+QPXtwDHiB1R/3t13JLW5GnicePfNvlS+WAdU0+tAWz/f3H6Mb758nJaOQSKhPG65uJZ3XbGYt19aT0mB+udFslGq3TIpjXM3s3cAfw6EgL919/9tZp8Gtrj7JjP7HnA5cCLxI0fcfcPZPlPhPj/cnZeOdvOtV07wr6+c4GTvMJH8PN5yUQ0/s3YRt11aT1VxJNNlikiK5jTc00HhPv8mJpxtR7r49qsn+c6rJznWPUSewbrmKm6/tJ7b1tSzvKY402WKyFko3OWs3J0dx3t5asdJvrvzFLtP9gGwvKaYWy+u49ZLarmuuYrCsM6IFVlIFO5yTo52DvLM7tM8s/s0zx/sIDY2QWE4jxtXVHPz6lresqqWlbXFGmIpkmEKdzlvQ7Fxnj/YznN72/n3vW0cStxMZHF5IW+6qIabVlZz08oaFpVrmKXIfFO4y5w52jnID/a188P9bfz4QAfdg/GbiqyoLeaGFdXxx/IqjakXmQcKd0mLiQln54lenj/QwY8PtLP5cNfUzb6bq6OsX17Fdc3xR1N1VN04InNM4S7zYmx8gp0nennhYAc/OdTFlpbOqT37mpIC1jVVcm1TJdc0VXDZknIdoBW5QAp3yYiJCWff6X62tHSy9XAXW1q6ONI5CEAklMeaJWVctbSCq5dVcNXSCpZVae9e5Fwo3GXBON03zPYj3Ww70sX2I938tLWHodFxACqjYa5orODKxnIub6zgisZy6tV3LzIjhbssWGPjE+w+2ccrrT28fLSbl1u72Xuqj4nEr2JtaQGXN5SzdkkZaxvKuayhnCXlhdrDF2FurwopMqfyQ3msbShnbUM5P3/9MgAGY2PsOtHLK609/LS1h1eP9/DsntNTgV8RDXPZkjLWLC7j0sTjoroSwiHdBlhkOgp3WRCikXyubari2qaqqXlDsXF2nexlx/Fedh7v4dVjvXzp+RZiY/EbhodDxkV1pVy6qJRLFpdy8aIyLllUSl1pgfbyJecp3GXBKoqEuGZZJdcsq5yaNzY+waH2AXae6GXXiT52nejlRwfa+fr2Y1NtKqJhVteXcnF9KavrS1hVX8rq+lJdIE1yisJdskp+KI9V9aWsqi/lrqtem981EGP3yT72nOxlz6l+9p7q4xvbj9GXGIMPUF0c4aK6ElbVl3BRbQkX1ZVyUV0J9WXa05fgUbhLIFQWR7hxZTU3rqyemufunOwdZu+pfvad6mPfqX72ne7jmy8dp2/4tdAvKchnZW0xK2tLWFlXwoqaYlbUltBUHdW4fMlaCncJLDNjcXkRi8uLuGV17dR8d6etb4T9p/vZ39bPgdP9HGgb4McHOl7XvWMGDRVFrKiNB35zdZTmmmJW1JSwpKKQfB3MlQVM4S45x8yoKyukrqyQmy6qed17AyNjHGof4EBbPwfbBjjUPsDB9n62tbx2mQWIH8xdWhmlqTpKU3U8+JtqimmqitJYGSWSr+CXzFK4iyQpLsifGqaZzN1p6x/hcPsgh9sHONQxEH9uH+DFQ50Mxsan2uYZLKkoYllVPPyXVkVZlvQoLwqrj1/STuEukgIzo660kLrSQtYvr3rde+5Oe3+Mlo4BWjoGaekcpKVjgCOdg3x35yna+2Ova19akE9jVZSllUUsTTw3VsY3Ag2VRbq/rcwJ/RaJXCAzo7a0gNrSAtY1V73h/YGRMY52DXKkY5AjnYO0dg1xpHOQg+0DPLevjeHRide1r4iGaagoorGyiIaKeOC/Nl1ERVR7/jI7hbtImhUX5HPJojIuWVT2hvfcnY6BGEcToX+se4ijnYMc6x7iYNsAP9jX/rouH4CicIglFYUsqYiH/eLyoqnpxeXxZ43yEYW7SAaZGTUlBdSUFHB10slak9yd7sFRjnUP0do1xPHu+AbgWNcQJ3qG2HWij/b+kTf8XGU0nBgpVMjiikIWlxexqKyQxeWFLEo8ohH9+QeZ1q7IAmZmVBZHqCyOvOEg76Th0XFO9Q5zvHuYEz1DnOgZ5nh3/PlY9xBbj3RNXWM/WWlhPovLC6kvK2RRWTzwJ1/XlxVSX1ZAdUkBoTx1AWUjhbtIlisMh2iqLqapunjGNkOxcU70DHGyd5hTvcOc6BnmVE/iuXeYvaf6aOsbmbpQ26RQnlFbUkB9WQG1pfHAry8rpK60gLqygvhB5rICqou1EVhoFO4iOaAoEoqfjFVbMmObsfEJ2vtjnEpsAOKPkfhz3witXYNsO9JF50DsDT+bZ1BdUkBd4sDy5HNtSXyjMHnAuba0gOJISAeE54HCXUSA+HV7Jvvjz2ZkbHxqI3C6d4S2vmFO943EX/ePcLpvmJ3He+kYiDF+5r8CQGE4j5qSeNBPHm+oLYlQkzRdUxKhuqSAssJ8bQjOk8JdRM5JQX6IhsRInbMZn3C6BmOc7h2hvT/+aOuLP9r74xuCIx2DbGvponMwxnT3DYqE8qguiVBTUkB1SYTq4njwVxXHw7+6OEL15HRxAUURjRKapHAXkbQI5b02Emg2Y+MTdA7GaO+LTW0IOvpjtA+M0N4Xo3NghPb+GHtO9tExEJu6pv+ZopFQIujjgV859brgdfOqiiNURSOUFuaTF9BjBQp3Ecm4/FDe1BnAs3F3BmLjtPeN0DEQo3MgRkf/a687B2J0DMRo6x+Z2hiMzLAxCOUZldEwldFE6CeeK6NhqoojVETjryuiiQ1DNExZYTgrNggphbuZ3QH8BRACHnb3Pz7j/ZuBPweuAO5298fnulAREYgPDy0pyKekIJ/mmplHCE1yd4ZGx+noj9E1+NoGoHPgtemugVE6B2McaOunqyVG1+DotMcLIH7wuLwovkEoT2wYKoriG4CKaHhqY1ARDVNRFH8uj4YpLZjf4wezhruZhYAHgduBVmCzmW1y951JzY4AHwL+ezqKFBE5X2ZGNJJPtCqfpVXRlH7G3ekdHqN7MB70XYMxugbir+PzYnQPjtI9OMqp3mH2nOyjezDGwBlnEycL5RnlRWHKi8L8xu2r2XDlkrlaxGmlsue+Htjv7gcBzOxR4C5gKtzd/XDiven/9xERySJmrwVxU/Xs7SfFxiboHorRMzhK1+AoPUPxjUH35OuhGD1DY1RF03/Lx1TCvQE4mjTdClx/Pl9mZhuBjQDLli07n48QEVmwIvmpHztIt1TuKDBdJ9H0nVGzcPeH3H2du6+rra2d/QdEROS8pBLurcDSpOlG4Hh6yhERkbmQSrhvBlaZ2XIziwB3A5vSW5aIiFyIWcPd3ceAe4GngF3AY+6+w8w+bWYbAMzsOjNrBf4L8NdmtiOdRYuIyNmlNM7d3Z8Enjxj3gNJrzcT764REZEFQLdoFxEJIIW7iEgAKdxFRALIfLrrbM7HF5u1AS3n+eM1QPsclpMtcnG5c3GZITeXOxeXGc59uZvcfdYThTIW7hfCzLa4+7pM1zHfcnG5c3GZITeXOxeXGdK33OqWEREJIIW7iEgAZWu4P5TpAjIkF5c7F5cZcnO5c3GZIU3LnZV97iIicnbZuucuIiJnkXXhbmZ3mNkeM9tvZvdnup50MLOlZvZ9M9tlZjvM7OOJ+VVm9l0z25d4rsx0rXPNzEJmtt3MvpWYXm5mLyaW+Z8SF68LFDOrMLPHzWx3Yp3fmCPr+jcSv9+vmtk/mllh0Na3mf2tmZ02s1eT5k27bi3uLxPZ9oqZXXMh351V4Z50y787gTXAB8xsTWarSosx4JPufilwA/CrieW8H3ja3VcBTyemg+bjxC9QN+kzwOcSy9wF/EpGqkqvvwC+4+6XAFcSX/5Ar2szawDuA9a5+1ri92e+m+Ct778D7jhj3kzr9k5gVeKxEfjChXxxVoU7Sbf8c/cYMHnLv0Bx9xPuvi3xuo/4H3sD8WX9UqLZl4D3ZKbC9DCzRuCdwMOJaQPeBkzecD2Iy1wG3Ax8EcDdY+7eTcDXdUI+UGRm+UAUOEHA1re7Pwd0njF7pnV7F/Blj3sBqDCzxef73dkW7tPd8q8hQ7XMCzNrBq4GXgTq3f0ExDcAQF3mKkuLPwf+BzB5L95qoDtx2WkI5vpeAbQBjyS6ox42s2ICvq7d/Rjwp8AR4qHeA2wl+OsbZl63c5pv2Rbuc3bLv2xgZiXA14Bfd/feTNeTTmb2LuC0u29Nnj1N06Ct73zgGuAL7n41MEDAumCmk+hnvgtYDiwBiol3S5wpaOv7bOb09z3bwj1nbvlnZmHiwf4P7v71xOxTk/+mJZ5PZ6q+NHgTsMHMDhPvbnsb8T35isS/7RDM9d0KtLr7i4npx4mHfZDXNcBtwCF3b3P3UeDrwE0Ef33DzOt2TvMt28I9J275l+hr/iKwy90/m/TWJuCexOt7gG/Od23p4u6fcvdGd28mvl6fcfdfAL4PvC/RLFDLDODuJ4GjZnZxYtbbgZ0EeF0nHAFuMLNo4vd9crkDvb4TZlq3m4BfSoyauQHomey+OS/unlUP4B3AXuAA8NuZridNy/hm4v+OvQK8lHi8g3gf9NPAvsRzVaZrTdPyvxX4VuL1CuAnwH7gn4GCTNeXhuW9CtiSWN/fACpzYV0DfwDsBl4FvgIUBG19A/9I/JjCKPE981+Zad0S75Z5MJFtPyU+kui8v1tnqIqIBFC2dcuIiEgKFO4iIgGkcBcRCSCFu4hIACncRUQCSOEuIhJACncRkQBSuIuIBNB/AGvLpKMXCKFDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(closses);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Questions:\n",
    "\n",
    "  - Why is the gradient going always down?\n",
    "  - How to check when to stop the learning process?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Gradient descent in NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The goal is to train a model that will distinguish Spanish and English names (feel free to do it with other languages).\n",
    "\n",
    "I'll use the data from [this website](https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.151781Z",
     "start_time": "2019-05-03T11:47:22.064449Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "head: cannot open './names/English.txt' for reading: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!head ./names/English.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.222309Z",
     "start_time": "2019-05-03T11:47:22.156704Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "head: cannot open './names/Spanish.txt' for reading: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!head ./names/Spanish.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.549072Z",
     "start_time": "2019-05-03T11:47:22.226672Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: './/names/English.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-10-5b31b38a1a7d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mf\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\".//names/English.txt\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mnames_english\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mline\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"./names/Spanish.txt\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mnames_spanish\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mline\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstrip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './/names/English.txt'"
     ]
    }
   ],
   "source": [
    "f=open(\".//names/English.txt\")\n",
    "names_english=[line.strip().lower() for line in f]\n",
    "f.close()\n",
    "f=open(\"./names/Spanish.txt\")\n",
    "names_spanish=[line.strip().lower() for line in f]\n",
    "f.close()\n",
    "len(names_english),names_english[:5],len(names_spanish),names_spanish[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:48:05.369517Z",
     "start_time": "2019-05-03T11:47:58.860754Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "#### Represent these names as Bag of Character bigrams\n",
    "\n",
    "#### and English/Spanish as classes 0 and 1\n",
    "\n",
    "#### save them in variables `X_train` and `y_train`\n",
    "\n",
    "#### and use the following code to find good parameters\n",
    "\n",
    "##### Hint: `CountVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.553998Z",
     "start_time": "2019-05-03T11:47:19.846Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.555982Z",
     "start_time": "2019-05-03T11:47:19.851Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#MODEL SPECIFICATION\n",
    "model = dy.ParameterCollection()\n",
    "\n",
    "W_p = model.add_parameters((2, 536))\n",
    "b_p = model.add_parameters (2)\n",
    "trainer = dy.SimpleSGDTrainer (model)\n",
    "\n",
    "num_epochs = 100\n",
    "closses = []\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    closs = 0\n",
    "\n",
    "    for input_x, out_label in zip(X_train,y_train):\n",
    "        dy.renew_cg()\n",
    "        x = dy.inputVector(input_x.toarray().reshape(-1)) #This must go after renewing the graph\n",
    "        \n",
    "        output_value = dy.logistic(W_p*x+b_p)\n",
    "        \n",
    "        loss = dy.hinge(output_value,out_label)\n",
    "\n",
    "        loss_val = loss.value()\n",
    "        closs = closs + loss_val\n",
    "        \n",
    "        loss.backward ()\n",
    "        trainer.update()\n",
    "    \n",
    "    print (closs/len(y_train))\n",
    "    closses.append(closs/len(y_train))\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.557471Z",
     "start_time": "2019-05-03T11:47:19.855Z"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(closses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Intuition about verbs in Spanish (blackboard)\n",
    "\n",
    "  - tocar $\\rightarrow$ ?\n",
    "  - Extra raw text\n",
    "  - Other possible usages of raw text\n",
    "    - POS tagging + extra character information (`frontofning`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.559457Z",
     "start_time": "2019-05-03T11:47:19.873Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!cp ../experiments/util.py ./\n",
    "!cp ../experiments/allnames.txt ./\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.560944Z",
     "start_time": "2019-05-03T11:47:19.879Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import util\n",
    "from itertools import count\n",
    "import random\n",
    "import time\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.562977Z",
     "start_time": "2019-05-03T11:47:19.884Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class RNNLanguageModel:\n",
    "    def __init__(self, model, LAYERS, INPUT_DIM, HIDDEN_DIM, VOCAB_SIZE, builder=dy.SimpleRNNBuilder):\n",
    "        self.builder = builder(LAYERS, INPUT_DIM, HIDDEN_DIM, model)\n",
    "\n",
    "        self.lookup = model.add_lookup_parameters((VOCAB_SIZE, INPUT_DIM), name=\"lookup\")\n",
    "        self.R = model.add_parameters((VOCAB_SIZE, HIDDEN_DIM), name=\"hidden2out\")\n",
    "        self.bias = model.add_parameters((VOCAB_SIZE), name=\"bias\")\n",
    "\n",
    "    def save_to_disk(self, filename):\n",
    "        dy.save(filename, [self.builder, self.lookup, self.R, self.bias])\n",
    "\n",
    "    def load_from_disk(self, filename):\n",
    "        (self.builder, self.lookup, self.R, self.bias) = dy.load(filename, model)\n",
    "        \n",
    "    def build_lm_graph(self, sent):\n",
    "        dy.renew_cg()\n",
    "        init_state = self.builder.initial_state()\n",
    "\n",
    "        errs = [] # will hold expressions\n",
    "        es=[]\n",
    "        state = init_state\n",
    "        for (cw,nw) in zip(sent,sent[1:]):\n",
    "            # assume word is already a word-id\n",
    "            x_t = dy.lookup(self.lookup, int(cw))\n",
    "            state = state.add_input(x_t)\n",
    "            y_t = state.output()\n",
    "            r_t = self.bias + (self.R * y_t)\n",
    "            err = dy.pickneglogsoftmax(r_t, int(nw))\n",
    "            errs.append(err)\n",
    "        nerr = dy.esum(errs)\n",
    "        return nerr\n",
    "    \n",
    "    def predict_next_word(self, sentence):\n",
    "        dy.renew_cg()\n",
    "        init_state = self.builder.initial_state()\n",
    "        state = init_state\n",
    "        for cw in sentence:\n",
    "            # assume word is already a word-id\n",
    "            x_t = self.lookup[int(cw)]\n",
    "            state = state.add_input(x_t)\n",
    "        y_t = state.output()\n",
    "        r_t = self.bias + (self.R * y_t)\n",
    "        prob = dy.softmax(r_t)\n",
    "        return prob\n",
    "        \n",
    "    def sample(self, first=1, nchars=0, stop=-1):\n",
    "        res = [first]\n",
    "        dy.renew_cg()\n",
    "        state = self.builder.initial_state()\n",
    "\n",
    "        cw = first\n",
    "        while True:\n",
    "            x_t = self.lookup[cw]\n",
    "            state = state.add_input(x_t)\n",
    "            y_t = state.output()\n",
    "            r_t = self.bias + (self.R * y_t)\n",
    "            ydist = dy.softmax(r_t)\n",
    "            dist = ydist.vec_value()\n",
    "            rnd = random.random()\n",
    "            for i,p in enumerate(dist):\n",
    "                rnd -= p\n",
    "                if rnd <= 0: break\n",
    "            res.append(i)\n",
    "            cw = i\n",
    "            if cw == stop: break\n",
    "            if nchars and len(res) > nchars: break\n",
    "        return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.564565Z",
     "start_time": "2019-05-03T11:47:19.889Z"
    }
   },
   "outputs": [],
   "source": [
    "help(util.CharsCorpusReader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-03T11:47:22.565409Z",
     "start_time": "2019-05-03T11:47:19.893Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "corpus = \"allnames.txt\"\n",
    "\n",
    "LAYERS = 2\n",
    "INPUT_DIM = 32 #50  #256\n",
    "HIDDEN_DIM = 128 # 50  #1024\n",
    "\n",
    "train = util.CharsCorpusReader(corpus, begin=\"<s>\")\n",
    "vocab = util.Vocab.from_corpus(train)\n",
    "\n",
    "VOCAB_SIZE = vocab.size()\n",
    "\n",
    "model = dy.Model()\n",
    "\n",
    "trainer = dy.SimpleSGDTrainer(model, learning_rate=0.2)\n",
    "\n",
    "lm = RNNLanguageModel(model, LAYERS, INPUT_DIM, HIDDEN_DIM, VOCAB_SIZE, builder=dy.SimpleRNNBuilder)\n",
    "#lm = RNNLanguageModel(model, LAYERS, INPUT_DIM, HIDDEN_DIM, VOCAB_SIZE, builder=dy.LSTMBuilder)\n",
    "\n",
    "\n",
    "train = list(train)\n",
    "\n",
    "losses = []\n",
    "\n",
    "chars = loss = 0.0\n",
    "for ITER in range(3):\n",
    "    random.shuffle(train)\n",
    "    for i,sent in enumerate(train):\n",
    "        _start = time.time()\n",
    "        if i % 2500 == 0:\n",
    "            trainer.status()\n",
    "            print (i,len(train))\n",
    "            if chars > 0: print(loss / chars,)\n",
    "            for _ in range(1):\n",
    "                samp = lm.sample(first=vocab.w2i[\"<s>\"],stop=vocab.w2i[\"\\n\"])\n",
    "                print(\"\".join([vocab.i2w[c] for c in samp]).strip())\n",
    "            loss = 0.0\n",
    "            chars = 0.0\n",
    "\n",
    "        chars += len(sent)-1\n",
    "        isent = [vocab.w2i[w] for w in sent]\n",
    "        errs = lm.build_lm_graph(isent)\n",
    "        loss += errs.scalar_value()\n",
    "        errs.backward()\n",
    "        trainer.update()\n",
    "    print (\"TM:\",(time.time() - _start)/len(sent))\n",
    "    print(\"ITER {}, loss={}\".format(ITER, loss))\n",
    "    losses.append(loss)\n",
    "    trainer.status()\n",
    "\n",
    "lm.save_to_disk(\"RNNLanguageModel.model\")\n",
    "\n",
    "print(\"loading the saved model...\")\n",
    "lm.load_from_disk(\"RNNLanguageModel.model\")\n",
    "samp = lm.sample(first=vocab.w2i[\"<s>\"],stop=vocab.w2i[\"\\n\"])\n",
    "print(\"\".join([vocab.i2w[c] for c in samp]).strip())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Question:\n",
    "\n",
    "  - How would you get the character-based representation of a word, such as, `Peter` or `animalkind`?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}