Dinosaurus_Island_Character_level_language_model_final_v3b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qo4gVJ_D01SH"
   },
   "source": [
    "# Character level language model - Dinosaurus Island\n",
    "\n",
    "Welcome to Dinosaurus Island! 65 million years ago, dinosaurs existed, and in this assignment they are back. You are in charge of a special task. Leading biology researchers are creating new breeds of dinosaurs and bringing them to life on earth, and your job is to give names to these dinosaurs. If a dinosaur does not like its name, it might go berserk, so choose wisely! \n",
    "\n",
    "<table>\n",
    "<td>\n",
    "<img src=\"images/dino.jpg\" style=\"width:250;height:300px;\">\n",
    "\n",
    "</td>\n",
    "\n",
    "</table>\n",
    "\n",
    "Luckily you have learned some deep learning and you will use it to save the day. Your assistant has collected a list of all the dinosaur names they could find, and compiled them into this [dataset](dinos.txt). (Feel free to take a look by clicking the previous link.) To create new dinosaur names, you will build a character level language model to generate new names. Your algorithm will learn the different name patterns, and randomly generate new names. Hopefully this algorithm will keep you and your team safe from the dinosaurs' wrath! \n",
    "\n",
    "By completing this assignment you will learn:\n",
    "\n",
    "- How to store text data for processing using an RNN \n",
    "- How to synthesize data, by sampling predictions at each time step and passing it to the next RNN-cell unit\n",
    "- How to build a character-level text generation recurrent neural network\n",
    "- Why clipping the gradients is important\n",
    "\n",
    "We will begin by loading in some functions that we have provided for you in `rnn_utils`. Specifically, you have access to functions such as `rnn_forward` and `rnn_backward` which are equivalent to those you've implemented in the previous assignment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Be6_xniP01SI"
   },
   "source": [
    "## <font color='darkblue'>Updates</font>\n",
    "\n",
    "#### If you were working on the notebook before this update...\n",
    "* The current notebook is version \"3b\".\n",
    "* You can find your original work saved in the notebook with the previous version name (\"v3a\") \n",
    "* To view the file directory, go to the menu \"File->Open\", and this will open a new tab that shows the file directory.\n",
    "\n",
    "#### List of updates 3b\n",
    "- removed redundant numpy import\n",
    "* `clip`\n",
    "    - change test code to use variable name 'mvalue' rather than 'maxvalue' and deleted it from namespace to avoid confusion.\n",
    "* `optimize`\n",
    "    - removed redundant description of clip function to discourage use of using 'maxvalue' which is not an argument to optimize\n",
    "* `model`\n",
    "    - added 'verbose mode to print X,Y to aid in creating that code.\n",
    "    - wordsmith instructions to prevent confusion\n",
    "        - 2000 examples vs 100, 7 displayed vs 10\n",
    "        - no randomization of order\n",
    "* `sample`\n",
    "    - removed comments regarding potential different sample outputs to reduce confusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "r0Nj4psY01SJ"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from utils import *\n",
    "import random\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h_elAxqq01SN"
   },
   "source": [
    "## 1 - Problem Statement\n",
    "\n",
    "### 1.1 - Dataset and Preprocessing\n",
    "\n",
    "Run the following cell to read the dataset of dinosaur names, create a list of unique characters (such as a-z), and compute the dataset and vocabulary size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qB2XWVg_01SO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 19909 total characters and 27 unique characters in your data.\n"
     ]
    }
   ],
   "source": [
    "data = open('dinos.txt', 'r').read()\n",
    "data= data.lower()\n",
    "chars = list(set(data))\n",
    "data_size, vocab_size = len(data), len(chars)\n",
    "print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CfmL668r01SQ"
   },
   "source": [
    "\n",
    "* The characters are a-z (26 characters) plus the \"\\n\" (or newline character).\n",
    "* In this assignment, the newline character \"\\n\" plays a role similar to the `<EOS>` (or \"End of sentence\") token we had discussed in lecture.  \n",
    "    - Here, \"\\n\" indicates the end of the dinosaur name rather than the end of a sentence. \n",
    "* `char_to_ix`: In the cell below, we create a python dictionary (i.e., a hash table) to map each character to an index from 0-26.\n",
    "* `ix_to_char`: We also create a second python dictionary that maps each index back to the corresponding character. \n",
    "    -  This will help you figure out what index corresponds to what character in the probability distribution output of the softmax layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bh3QcYpr01SQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\n', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(chars)\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2YltsxeZ01SU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   0: '\\n',\n",
      "    1: 'a',\n",
      "    2: 'b',\n",
      "    3: 'c',\n",
      "    4: 'd',\n",
      "    5: 'e',\n",
      "    6: 'f',\n",
      "    7: 'g',\n",
      "    8: 'h',\n",
      "    9: 'i',\n",
      "    10: 'j',\n",
      "    11: 'k',\n",
      "    12: 'l',\n",
      "    13: 'm',\n",
      "    14: 'n',\n",
      "    15: 'o',\n",
      "    16: 'p',\n",
      "    17: 'q',\n",
      "    18: 'r',\n",
      "    19: 's',\n",
      "    20: 't',\n",
      "    21: 'u',\n",
      "    22: 'v',\n",
      "    23: 'w',\n",
      "    24: 'x',\n",
      "    25: 'y',\n",
      "    26: 'z'}\n"
     ]
    }
   ],
   "source": [
    "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(ix_to_char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OuDQfApB01SW"
   },
   "source": [
    "### 1.2 - Overview of the model\n",
    "\n",
    "Your model will have the following structure: \n",
    "\n",
    "- Initialize parameters \n",
    "- Run the optimization loop\n",
    "    - Forward propagation to compute the loss function\n",
    "    - Backward propagation to compute the gradients with respect to the loss function\n",
    "    - Clip the gradients to avoid exploding gradients\n",
    "    - Using the gradients, update your parameters with the gradient descent update rule.\n",
    "- Return the learned parameters \n",
    "    \n",
    "<img src=\"images/rnn.png\" style=\"width:450;height:300px;\">\n",
    "<caption><center> **Figure 1**: Recurrent Neural Network, similar to what you had built in the previous notebook \"Building a Recurrent Neural Network - Step by Step\".  </center></caption>\n",
    "\n",
    "* At each time-step, the RNN tries to predict what is the next character given the previous characters. \n",
    "* The dataset $\\mathbf{X} = (x^{\\langle 1 \\rangle}, x^{\\langle 2 \\rangle}, ..., x^{\\langle T_x \\rangle})$ is a list of characters in the training set.\n",
    "* $\\mathbf{Y} = (y^{\\langle 1 \\rangle}, y^{\\langle 2 \\rangle}, ..., y^{\\langle T_x \\rangle})$ is the same list of characters but shifted one character forward. \n",
    "* At every time-step $t$, $y^{\\langle t \\rangle} = x^{\\langle t+1 \\rangle}$.  The prediction at time $t$ is the same as the input at time $t + 1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bNFjtuyq01SW"
   },
   "source": [
    "## 2 - Building blocks of the model\n",
    "\n",
    "In this part, you will build two important blocks of the overall model:\n",
    "- Gradient clipping: to avoid exploding gradients\n",
    "- Sampling: a technique used to generate characters\n",
    "\n",
    "You will then apply these two functions to build the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tuVLtQaM01SX"
   },
   "source": [
    "### 2.1 - Clipping the gradients in the optimization loop\n",
    "\n",
    "In this section you will implement the `clip` function that you will call inside of your optimization loop. \n",
    "\n",
    "#### Exploding gradients\n",
    "* When gradients are very large, they're called \"exploding gradients.\"  \n",
    "* Exploding gradients make the training process more difficult, because the updates may be so large that they \"overshoot\" the optimal values during back propagation.\n",
    "\n",
    "Recall that your overall loop structure usually consists of:\n",
    "* forward pass, \n",
    "* cost computation, \n",
    "* backward pass, \n",
    "* parameter update. \n",
    "\n",
    "Before updating the parameters, you will perform gradient clipping to make sure that your gradients are not \"exploding.\"\n",
    "\n",
    "#### gradient clipping\n",
    "In the exercise below, you will implement a function `clip` that takes in a dictionary of gradients and returns a clipped version of gradients if needed. \n",
    "* There are different ways to clip gradients.\n",
    "* We will use a simple element-wise clipping procedure, in which every element of the gradient vector is clipped to lie between some range [-N, N]. \n",
    "* For example, if the N=10\n",
    "    - The range is [-10, 10]\n",
    "    - If any component of the gradient vector is greater than 10, it is set to 10.\n",
    "    - If any component of the gradient vector is less than -10, it is set to -10. \n",
    "    - If any components are between -10 and 10, they keep their original values.\n",
    "\n",
    "<img src=\"images/clip.png\" style=\"width:400;height:150px;\">\n",
    "<caption><center> **Figure 2**: Visualization of gradient descent with and without gradient clipping, in a case where the network is running into \"exploding gradient\" problems. </center></caption>\n",
    "\n",
    "**Exercise**: \n",
    "Implement the function below to return the clipped gradients of your dictionary `gradients`. \n",
    "* Your function takes in a maximum threshold and returns the clipped versions of the gradients. \n",
    "* You can check out [numpy.clip](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.clip.html). \n",
    "    - You will need to use the argument \"`out = ...`\".\n",
    "    - Using the \"`out`\" parameter allows you to update a variable \"in-place\".\n",
    "    - If you don't use \"`out`\" argument, the clipped variable is stored in the variable \"gradient\" but does not update the gradient variables `dWax`, `dWaa`, `dWya`, `db`, `dby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "2yYvYeI501SX"
   },
   "outputs": [],
   "source": [
    "### GRADED FUNCTION: clip\n",
    "\n",
    "def clip(gradients, maxValue):\n",
    "    '''\n",
    "    Clips the gradients' values between minimum and maximum.\n",
    "    \n",
    "    Arguments:\n",
    "    gradients -- a dictionary containing the gradients \"dWaa\", \"dWax\", \"dWya\", \"db\", \"dby\"\n",
    "    maxValue -- everything above this number is set to this number, and everything less than -maxValue is set to -maxValue\n",
    "    \n",
    "    Returns: \n",
    "    gradients -- a dictionary with the clipped gradients.\n",
    "    '''\n",
    "    \n",
    "    dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']\n",
    "   \n",
    "    ### START CODE HERE ###\n",
    "    # clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)\n",
    "    for gradient in [dWax, dWaa, dWya, db, dby]:\n",
    "        gradient = np.clip(gradient, -maxValue, maxValue, out = gradient)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    gradients = {\"dWaa\": dWaa, \"dWax\": dWax, \"dWya\": dWya, \"db\": db, \"dby\": dby}\n",
    "    \n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NAdY1Eon01Sa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradients[\"dWaa\"][1][2] = 10.0\n",
      "gradients[\"dWax\"][3][1] = -10.0\n",
      "gradients[\"dWya\"][1][2] = 0.29713815361\n",
      "gradients[\"db\"][4] = [ 10.]\n",
      "gradients[\"dby\"][1] = [ 8.45833407]\n"
     ]
    }
   ],
   "source": [
    "# Test with a maxvalue of 10\n",
    "mValue = 10\n",
    "np.random.seed(3)\n",
    "dWax = np.random.randn(5,3)*10\n",
    "dWaa = np.random.randn(5,5)*10\n",
    "dWya = np.random.randn(2,5)*10\n",
    "db = np.random.randn(5,1)*10\n",
    "dby = np.random.randn(2,1)*10\n",
    "gradients = {\"dWax\": dWax, \"dWaa\": dWaa, \"dWya\": dWya, \"db\": db, \"dby\": dby}\n",
    "gradients = clip(gradients, mValue)\n",
    "print(\"gradients[\\\"dWaa\\\"][1][2] =\", gradients[\"dWaa\"][1][2])\n",
    "print(\"gradients[\\\"dWax\\\"][3][1] =\", gradients[\"dWax\"][3][1])\n",
    "print(\"gradients[\\\"dWya\\\"][1][2] =\", gradients[\"dWya\"][1][2])\n",
    "print(\"gradients[\\\"db\\\"][4] =\", gradients[\"db\"][4])\n",
    "print(\"gradients[\\\"dby\\\"][1] =\", gradients[\"dby\"][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "58jmsqzi01Sd"
   },
   "source": [
    "** Expected output:**\n",
    "\n",
    "```Python\n",
    "gradients[\"dWaa\"][1][2] = 10.0\n",
    "gradients[\"dWax\"][3][1] = -10.0\n",
    "gradients[\"dWya\"][1][2] = 0.29713815361\n",
    "gradients[\"db\"][4] = [ 10.]\n",
    "gradients[\"dby\"][1] = [ 8.45833407]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6O4BLjJf01Sd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradients[\"dWaa\"][1][2] = 5.0\n",
      "gradients[\"dWax\"][3][1] = -5.0\n",
      "gradients[\"dWya\"][1][2] = 0.29713815361\n",
      "gradients[\"db\"][4] = [ 5.]\n",
      "gradients[\"dby\"][1] = [ 5.]\n"
     ]
    }
   ],
   "source": [
    "# Test with a maxValue of 5\n",
    "mValue = 5\n",
    "np.random.seed(3)\n",
    "dWax = np.random.randn(5,3)*10\n",
    "dWaa = np.random.randn(5,5)*10\n",
    "dWya = np.random.randn(2,5)*10\n",
    "db = np.random.randn(5,1)*10\n",
    "dby = np.random.randn(2,1)*10\n",
    "gradients = {\"dWax\": dWax, \"dWaa\": dWaa, \"dWya\": dWya, \"db\": db, \"dby\": dby}\n",
    "gradients = clip(gradients, mValue)\n",
    "print(\"gradients[\\\"dWaa\\\"][1][2] =\", gradients[\"dWaa\"][1][2])\n",
    "print(\"gradients[\\\"dWax\\\"][3][1] =\", gradients[\"dWax\"][3][1])\n",
    "print(\"gradients[\\\"dWya\\\"][1][2] =\", gradients[\"dWya\"][1][2])\n",
    "print(\"gradients[\\\"db\\\"][4] =\", gradients[\"db\"][4])\n",
    "print(\"gradients[\\\"dby\\\"][1] =\", gradients[\"dby\"][1])\n",
    "del mValue # avoid common issue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tyO2dJow01Sf"
   },
   "source": [
    "** Expected Output: **\n",
    "```Python\n",
    "gradients[\"dWaa\"][1][2] = 5.0\n",
    "gradients[\"dWax\"][3][1] = -5.0\n",
    "gradients[\"dWya\"][1][2] = 0.29713815361\n",
    "gradients[\"db\"][4] = [ 5.]\n",
    "gradients[\"dby\"][1] = [ 5.]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cxSUDRYH01Sg"
   },
   "source": [
    "### 2.2 - Sampling\n",
    "\n",
    "Now assume that your model is trained. You would like to generate new text (characters). The process of generation is explained in the picture below:\n",
    "\n",
    "<img src=\"images/dinos3.png\" style=\"width:500;height:300px;\">\n",
    "<caption><center> **Figure 3**: In this picture, we assume the model is already trained. We pass in $x^{\\langle 1\\rangle} = \\vec{0}$ at the first time step, and have the network sample one character at a time. </center></caption>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2l1SKw3301Sg"
   },
   "source": [
    "**Exercise**: Implement the `sample` function below to sample characters. You need to carry out 4 steps:\n",
    "\n",
    "- **Step 1**: Input the \"dummy\" vector of zeros $x^{\\langle 1 \\rangle} = \\vec{0}$. \n",
    "    - This is the default input before we've generated any characters. \n",
    "    We also set $a^{\\langle 0 \\rangle} = \\vec{0}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2VAXHjEd01Sh"
   },
   "source": [
    "- **Step 2**: Run one step of forward propagation to get $a^{\\langle 1 \\rangle}$ and $\\hat{y}^{\\langle 1 \\rangle}$. Here are the equations:\n",
    "\n",
    "hidden state:  \n",
    "$$ a^{\\langle t+1 \\rangle} = \\tanh(W_{ax}  x^{\\langle t+1 \\rangle } + W_{aa} a^{\\langle t \\rangle } + b)\\tag{1}$$\n",
    "\n",
    "activation:\n",
    "$$ z^{\\langle t + 1 \\rangle } = W_{ya}  a^{\\langle t + 1 \\rangle } + b_y \\tag{2}$$\n",
    "\n",
    "prediction:\n",
    "$$ \\hat{y}^{\\langle t+1 \\rangle } = softmax(z^{\\langle t + 1 \\rangle })\\tag{3}$$\n",
    "\n",
    "- Details about $\\hat{y}^{\\langle t+1 \\rangle }$:\n",
    "   - Note that $\\hat{y}^{\\langle t+1 \\rangle }$ is a (softmax) probability vector (its entries are between 0 and 1 and sum to 1). \n",
    "   - $\\hat{y}^{\\langle t+1 \\rangle}_i$ represents the probability that the character indexed by \"i\" is the next character.  \n",
    "   - We have provided a `softmax()` function that you can use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4QkJOHAS01Sh"
   },
   "source": [
    "#### Additional Hints\n",
    "\n",
    "- $x^{\\langle 1 \\rangle}$ is `x` in the code. When creating the one-hot vector, make a numpy array of zeros, with the number of rows equal to the number of unique characters, and the number of columns equal to one.  It's a 2D and not a 1D array.\n",
    "- $a^{\\langle 0 \\rangle}$ is `a_prev` in the code.  It is a numpy array of zeros, where the number of rows is $n_{a}$, and number of columns is 1.  It is a 2D array as well.  $n_{a}$ is retrieved by getting the number of columns in $W_{aa}$ (the numbers need to match in order for the matrix multiplication $W_{aa}a^{\\langle t \\rangle}$ to work.\n",
    "- [numpy.dot](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html)\n",
    "- [numpy.tanh](https://docs.scipy.org/doc/numpy/reference/generated/numpy.tanh.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q2CxggTH01Si"
   },
   "source": [
    "#### Using 2D arrays instead of 1D arrays\n",
    "* You may be wondering why we emphasize that $x^{\\langle 1 \\rangle}$ and $a^{\\langle 0 \\rangle}$ are 2D arrays and not 1D vectors.\n",
    "* For matrix multiplication in numpy, if we multiply a 2D matrix with a 1D vector, we end up with with a 1D array.\n",
    "* This becomes a problem when we add two arrays where we expected them to have the same shape.\n",
    "* When two arrays with  a different number of dimensions are added together, Python \"broadcasts\" one across the other.\n",
    "* Here is some sample code that shows the difference between using a 1D and 2D array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-jVKdq7Q01Sj"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matrix1 \n",
      " [[1 1]\n",
      " [2 2]\n",
      " [3 3]] \n",
      "\n",
      "matrix2 \n",
      " [[0]\n",
      " [0]\n",
      " [0]] \n",
      "\n",
      "vector1D \n",
      " [1 1] \n",
      "\n",
      "vector2D \n",
      " [[1]\n",
      " [1]]\n"
     ]
    }
   ],
   "source": [
    "matrix1 = np.array([[1,1],[2,2],[3,3]]) # (3,2)\n",
    "matrix2 = np.array([[0],[0],[0]]) # (3,1) \n",
    "vector1D = np.array([1,1]) # (2,) \n",
    "vector2D = np.array([[1],[1]]) # (2,1)\n",
    "print(\"matrix1 \\n\", matrix1,\"\\n\")\n",
    "print(\"matrix2 \\n\", matrix2,\"\\n\")\n",
    "print(\"vector1D \\n\", vector1D,\"\\n\")\n",
    "print(\"vector2D \\n\", vector2D)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7tCj1vZ-01Sl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Multiply 2D and 1D arrays: result is a 1D array\n",
      " [2 4 6]\n",
      "Multiply 2D and 2D arrays: result is a 2D array\n",
      " [[2]\n",
      " [4]\n",
      " [6]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Multiply 2D and 1D arrays: result is a 1D array\\n\", \n",
    "      np.dot(matrix1,vector1D))\n",
    "print(\"Multiply 2D and 2D arrays: result is a 2D array\\n\", \n",
    "      np.dot(matrix1,vector2D))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sGquAzEU01Sn"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding (3 x 1) vector to a (3 x 1) vector is a (3 x 1) vector\n",
      " This is what we want here!\n",
      " [[2]\n",
      " [4]\n",
      " [6]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Adding (3 x 1) vector to a (3 x 1) vector is a (3 x 1) vector\\n\",\n",
    "      \"This is what we want here!\\n\", \n",
    "      np.dot(matrix1,vector2D) + matrix2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "erK4JXVS01Sp"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding a (3,) vector to a (3 x 1) vector\n",
      " broadcasts the 1D array across the second dimension\n",
      " Not what we want here!\n",
      " [[2 4 6]\n",
      " [2 4 6]\n",
      " [2 4 6]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Adding a (3,) vector to a (3 x 1) vector\\n\",\n",
    "      \"broadcasts the 1D array across the second dimension\\n\",\n",
    "      \"Not what we want here!\\n\",\n",
    "      np.dot(matrix1,vector1D) + matrix2\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8_GeJwei01Ss"
   },
   "source": [
    "- **Step 3**: Sampling: \n",
    "    - Now that we have $y^{\\langle t+1 \\rangle}$, we want to select the next letter in the dinosaur name. If we select the most probable, the model will always generate the same result given a starting letter. To make the results more interesting, we will use np.random.choice to select a next letter that is *likely*, but not always the same.\n",
    "    - Pick the next character's **index** according to the probability distribution specified by $\\hat{y}^{\\langle t+1 \\rangle }$. \n",
    "    - This means that if $\\hat{y}^{\\langle t+1 \\rangle }_i = 0.16$, you will pick the index \"i\" with 16% probability. \n",
    "    - Use [np.random.choice](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.random.choice.html).\n",
    "\n",
    "    Example of how to use `np.random.choice()`:\n",
    "    ```python\n",
    "    np.random.seed(0)\n",
    "    probs = np.array([0.1, 0.0, 0.7, 0.2])\n",
    "    idx = np.random.choice(range(len((probs)), p = probs)\n",
    "    ```\n",
    "    \n",
    "    - This means that you will pick the index (`idx`) according to the distribution: \n",
    "\n",
    "    $P(index = 0) = 0.1, P(index = 1) = 0.0, P(index = 2) = 0.7, P(index = 3) = 0.2$.\n",
    "\n",
    "    - Note that the value that's set to `p` should be set to a 1D vector.\n",
    "    - Also notice that $\\hat{y}^{\\langle t+1 \\rangle}$, which is `y` in the code, is a 2D array.\n",
    "    - Also notice, while in your implementation, the first argument to np.random.choice is just an ordered list [0,1,.., vocab_len-1], it is *Not* appropriate to use char_to_ix.values(). The *order* of values returned by a python dictionary .values() call will be the same order as they are added to the dictionary. The grader may have a different order when it runs your routine than when you run it in your notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ReKfIdP501Ss"
   },
   "source": [
    "##### Additional Hints\n",
    "- [range](https://docs.python.org/3/library/functions.html#func-range)\n",
    "- [numpy.ravel](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ravel.html) takes a multi-dimensional array and returns its contents inside of a 1D vector.\n",
    "```Python\n",
    "arr = np.array([[1,2],[3,4]])\n",
    "print(\"arr\")\n",
    "print(arr)\n",
    "print(\"arr.ravel()\")\n",
    "print(arr.ravel())\n",
    "```\n",
    "Output:\n",
    "```Python\n",
    "arr\n",
    "[[1 2]\n",
    " [3 4]]\n",
    "arr.ravel()\n",
    "[1 2 3 4]\n",
    "```\n",
    "\n",
    "- Note that `append` is an \"in-place\" operation.  In other words, don't do this:\n",
    "```Python\n",
    "fun_hobbies = fun_hobbies.append('learning')  ## Doesn't give you what you want\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KHJ4tfWY01Su"
   },
   "source": [
    "- **Step 4**: Update to $x^{\\langle t \\rangle }$ \n",
    "    - The last step to implement in `sample()` is to update the variable `x`, which currently stores $x^{\\langle t \\rangle }$, with the value of $x^{\\langle t + 1 \\rangle }$. \n",
    "    - You will represent $x^{\\langle t + 1 \\rangle }$ by creating a one-hot vector corresponding to the character that you have chosen as your prediction. \n",
    "    - You will then forward propagate $x^{\\langle t + 1 \\rangle }$ in Step 1 and keep repeating the process until you get a \"\\n\" character, indicating that you have reached the end of the dinosaur name. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qqdlzd0l01Su"
   },
   "source": [
    "##### Additional Hints\n",
    "- In order to reset `x` before setting it to the new one-hot vector, you'll want to set all the values to zero.\n",
    "    - You can either create a new numpy array: [numpy.zeros](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html)\n",
    "    - Or fill all values with a single number: [numpy.ndarray.fill](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.fill.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "UIkYdtBx01Su"
   },
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: sample\n",
    "\n",
    "def sample(parameters, char_to_ix, seed):\n",
    "    \"\"\"\n",
    "    Sample a sequence of characters according to a sequence of probability distributions output of the RNN\n",
    "\n",
    "    Arguments:\n",
    "    parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b. \n",
    "    char_to_ix -- python dictionary mapping each character to an index.\n",
    "    seed -- used for grading purposes. Do not worry about it.\n",
    "\n",
    "    Returns:\n",
    "    indices -- a list of length n containing the indices of the sampled characters.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Retrieve parameters and relevant shapes from \"parameters\" dictionary\n",
    "    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']\n",
    "    vocab_size = by.shape[0]\n",
    "    n_a = Waa.shape[1]\n",
    "    \n",
    "    ### START CODE HERE ###\n",
    "    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)\n",
    "    x = np.zeros((vocab_size,1))\n",
    "    # Step 1': Initialize a_prev as zeros (≈1 line)\n",
    "    a_prev = np.zeros((n_a,1))\n",
    "    \n",
    "    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)\n",
    "    indices = []\n",
    "    \n",
    "    # Idx is a flag to detect a newline character, we initialize it to -1\n",
    "    idx = -1 \n",
    "    \n",
    "    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append \n",
    "    # its index to \"indices\". We'll stop if we reach 50 characters (which should be very unlikely with a well \n",
    "    # trained model), which helps debugging and prevents entering an infinite loop. \n",
    "    counter = 0\n",
    "    newline_character = char_to_ix['\\n']\n",
    "    \n",
    "    while (idx != newline_character and counter != 50):\n",
    "        \n",
    "        # Step 2: Forward propagate x using the equations (1), (2) and (3)\n",
    "        a = np.tanh(np.dot(Wax,x) + np.dot(Waa,a_prev) + b)\n",
    "        z = np.dot(Wya,a) + by\n",
    "        y = softmax(z)\n",
    "        \n",
    "        # for grading purposes\n",
    "        np.random.seed(counter+seed) \n",
    "        \n",
    "        # Step 3: Sample the index of a character within the vocabulary from the probability distribution y\n",
    "        idx = np.random.choice(list(range(vocab_size)), p = y.ravel())\n",
    "\n",
    "        # Append the index to \"indices\"\n",
    "        indices.append(idx)\n",
    "        \n",
    "        # Step 4: Overwrite the input character as the one corresponding to the sampled index.\n",
    "        x = np.zeros((vocab_size, 1))\n",
    "        x[idx] = 1\n",
    "        \n",
    "        # Update \"a_prev\" to be \"a\"\n",
    "        a_prev = a\n",
    "        \n",
    "        # for grading purposes\n",
    "        seed += 1\n",
    "        counter +=1\n",
    "        \n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    if (counter == 50):\n",
    "        indices.append(char_to_ix['\\n'])\n",
    "    \n",
    "    return indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Db8r6cXp01Sy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sampling:\n",
      "list of sampled indices:\n",
      " [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24, 7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 17, 24, 12, 13, 24, 0]\n",
      "list of sampled characters:\n",
      " ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o', 'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o', 'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'q', 'x', 'l', 'm', 'x', '\\n']\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(2)\n",
    "_, n_a = 20, 100\n",
    "Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)\n",
    "b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)\n",
    "parameters = {\"Wax\": Wax, \"Waa\": Waa, \"Wya\": Wya, \"b\": b, \"by\": by}\n",
    "\n",
    "\n",
    "indices = sample(parameters, char_to_ix, 0)\n",
    "print(\"Sampling:\")\n",
    "print(\"list of sampled indices:\\n\", indices)\n",
    "print(\"list of sampled characters:\\n\", [ix_to_char[i] for i in indices])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "emrm1Ud901S1"
   },
   "source": [
    "** Expected output:**\n",
    "\n",
    "```Python\n",
    "Sampling:\n",
    "list of sampled indices:\n",
    " [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24, 7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 17, 24, 12, 13, 24, 0]\n",
    "list of sampled characters:\n",
    " ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o', 'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o', 'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'q', 'x', 'l', 'm', 'x', '\\n']\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qdeVJ9xT01S2"
   },
   "source": [
    "## 3 - Building the language model \n",
    "\n",
    "It is time to build the character-level language model for text generation. \n",
    "\n",
    "\n",
    "### 3.1 - Gradient descent \n",
    "\n",
    "* In this section you will implement a function performing one step of stochastic gradient descent (with clipped gradients). \n",
    "* You will go through the training examples one at a time, so the optimization algorithm will be stochastic gradient descent. \n",
    "\n",
    "As a reminder, here are the steps of a common optimization loop for an RNN:\n",
    "\n",
    "- Forward propagate through the RNN to compute the loss\n",
    "- Backward propagate through time to compute the gradients of the loss with respect to the parameters\n",
    "- Clip the gradients\n",
    "- Update the parameters using gradient descent \n",
    "\n",
    "**Exercise**: Implement the optimization process (one step of stochastic gradient descent). \n",
    "\n",
    "The following functions are provided:\n",
    "\n",
    "```python\n",
    "def rnn_forward(X, Y, a_prev, parameters):\n",
    "    \"\"\" Performs the forward propagation through the RNN and computes the cross-entropy loss.\n",
    "    It returns the loss' value as well as a \"cache\" storing values to be used in backpropagation.\"\"\"\n",
    "    ....\n",
    "    return loss, cache\n",
    "    \n",
    "def rnn_backward(X, Y, parameters, cache):\n",
    "    \"\"\" Performs the backward propagation through time to compute the gradients of the loss with respect\n",
    "    to the parameters. It returns also all the hidden states.\"\"\"\n",
    "    ...\n",
    "    return gradients, a\n",
    "\n",
    "def update_parameters(parameters, gradients, learning_rate):\n",
    "    \"\"\" Updates parameters using the Gradient Descent Update Rule.\"\"\"\n",
    "    ...\n",
    "    return parameters\n",
    "```\n",
    "\n",
    "Recall that you previously implemented the `clip` function:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YsaPkvUq01S3"
   },
   "source": [
    "#### parameters\n",
    "\n",
    "* Note that the weights and biases inside the `parameters` dictionary are being updated by the optimization, even though `parameters` is not one of the returned values of the `optimize` function. The `parameters` dictionary is passed by reference into the function, so changes to this dictionary are making changes to the `parameters` dictionary even when accessed outside of the function.\n",
    "* Python dictionaries and lists are \"pass by reference\", which means that if you pass a dictionary into a function and modify the dictionary within the function, this changes that same dictionary (it's not a copy of the dictionary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "_BbEdIgY01S3"
   },
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: optimize\n",
    "\n",
    "def optimize(X, Y, a_prev, parameters, learning_rate = 0.01):\n",
    "    \"\"\"\n",
    "    Execute one step of the optimization to train the model.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- list of integers, where each integer is a number that maps to a character in the vocabulary.\n",
    "    Y -- list of integers, exactly the same as X but shifted one index to the left.\n",
    "    a_prev -- previous hidden state.\n",
    "    parameters -- python dictionary containing:\n",
    "                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)\n",
    "                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)\n",
    "                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)\n",
    "                        b --  Bias, numpy array of shape (n_a, 1)\n",
    "                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)\n",
    "    learning_rate -- learning rate for the model.\n",
    "    \n",
    "    Returns:\n",
    "    loss -- value of the loss function (cross-entropy)\n",
    "    gradients -- python dictionary containing:\n",
    "                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)\n",
    "                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)\n",
    "                        dWya -- Gradients of hidden-to-output weights, of shape (n_y, n_a)\n",
    "                        db -- Gradients of bias vector, of shape (n_a, 1)\n",
    "                        dby -- Gradients of output bias vector, of shape (n_y, 1)\n",
    "    a[len(X)-1] -- the last hidden state, of shape (n_a, 1)\n",
    "    \"\"\"\n",
    "    \n",
    "    #### START CODE HERE ###\n",
    "    \n",
    "    # Forward propagate through time (≈1 line)\n",
    "    loss, cache = rnn_forward(X,Y,a_prev,parameters)\n",
    "    \n",
    "    # Backpropagate through time (≈1 line)\n",
    "    gradients, a = rnn_backward(X,Y,parameters, cache)\n",
    "    \n",
    "    # Clip your gradients between -5 (min) and 5 (max) (≈1 line)\n",
    "    gradients = clip(gradients,5)\n",
    "    \n",
    "    # Update parameters (≈1 line)\n",
    "    parameters = update_parameters(parameters,gradients,learning_rate)\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    \n",
    "    return loss, gradients, a[len(X)-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ww-EKK5801S5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss = 126.503975722\n",
      "gradients[\"dWaa\"][1][2] = 0.194709315347\n",
      "np.argmax(gradients[\"dWax\"]) = 93\n",
      "gradients[\"dWya\"][1][2] = -0.007773876032\n",
      "gradients[\"db\"][4] = [-0.06809825]\n",
      "gradients[\"dby\"][1] = [ 0.01538192]\n",
      "a_last[4] = [-1.]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "vocab_size, n_a = 27, 100\n",
    "a_prev = np.random.randn(n_a, 1)\n",
    "Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)\n",
    "b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)\n",
    "parameters = {\"Wax\": Wax, \"Waa\": Waa, \"Wya\": Wya, \"b\": b, \"by\": by}\n",
    "X = [12,3,5,11,22,3]\n",
    "Y = [4,14,11,22,25, 26]\n",
    "\n",
    "loss, gradients, a_last = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)\n",
    "print(\"Loss =\", loss)\n",
    "print(\"gradients[\\\"dWaa\\\"][1][2] =\", gradients[\"dWaa\"][1][2])\n",
    "print(\"np.argmax(gradients[\\\"dWax\\\"]) =\", np.argmax(gradients[\"dWax\"]))\n",
    "print(\"gradients[\\\"dWya\\\"][1][2] =\", gradients[\"dWya\"][1][2])\n",
    "print(\"gradients[\\\"db\\\"][4] =\", gradients[\"db\"][4])\n",
    "print(\"gradients[\\\"dby\\\"][1] =\", gradients[\"dby\"][1])\n",
    "print(\"a_last[4] =\", a_last[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0r7EJrT801S8"
   },
   "source": [
    "** Expected output:**\n",
    "\n",
    "```Python\n",
    "Loss = 126.503975722\n",
    "gradients[\"dWaa\"][1][2] = 0.194709315347\n",
    "np.argmax(gradients[\"dWax\"]) = 93\n",
    "gradients[\"dWya\"][1][2] = -0.007773876032\n",
    "gradients[\"db\"][4] = [-0.06809825]\n",
    "gradients[\"dby\"][1] = [ 0.01538192]\n",
    "a_last[4] = [-1.]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m-4cL3oZ01S9"
   },
   "source": [
    "### 3.2 - Training the model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qZ63mN5-01S9"
   },
   "source": [
    "* Given the dataset of dinosaur names, we use each line of the dataset (one name) as one training example. \n",
    "* Every 2000 steps of stochastic gradient descent, you will sample several randomly chosen names to see how the algorithm is doing. \n",
    " \n",
    "\n",
    "**Exercise**: Follow the instructions and implement `model()`. When `examples[index]` contains one dinosaur name (string), to create an example (X, Y), you can use this:\n",
    "\n",
    "##### Set the index `idx` into the list of examples\n",
    "* Using the for-loop, walk through the shuffled list of dinosaur names in the list \"examples\".\n",
    "* For example, if there are n_e examples, and the for-loop increments the index to n_e onwards, think of how you would make the index cycle back to 0, so that we can continue feeding the examples into the model when j is n_e, n_e + 1, etc.\n",
    "* Hint: n_e + 1 divided by n_e is zero with a remainder of 1.\n",
    "* `%` is the modulus operator in python.\n",
    "\n",
    "##### Extract a single example from the list of examples\n",
    "* `single_example`: use the `idx` index that you set previously to get one word from the list of examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RNDsHeoQ01S9"
   },
   "source": [
    "##### Convert a string into a list of characters: `single_example_chars`\n",
    "* `single_example_chars`: A string is a list of characters.\n",
    "* You can use a list comprehension (recommended over for-loops) to generate a list of characters.\n",
    "```Python\n",
    "str = 'I love learning'\n",
    "list_of_chars = [c for c in str]\n",
    "print(list_of_chars)\n",
    "```\n",
    "\n",
    "```\n",
    "['I', ' ', 'l', 'o', 'v', 'e', ' ', 'l', 'e', 'a', 'r', 'n', 'i', 'n', 'g']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jzSvzjde01S-"
   },
   "source": [
    "##### Convert list of characters to a list of integers: `single_example_ix`\n",
    "* Create a list that contains the index numbers associated with each character.\n",
    "* Use the dictionary `char_to_ix`\n",
    "* You can combine this with the list comprehension that is used to get a list of characters from a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lL05MmsV01S-"
   },
   "source": [
    "##### Create the list of input characters: `X`\n",
    "* `rnn_forward` uses the **`None`** value as a flag to set the input vector as a zero-vector.\n",
    "* Prepend the list [**`None`**] in front of the list of input characters.\n",
    "* There is more than one way to prepend a value to a list.  One way is to add two lists together: `['a'] + ['b']`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2D8OXGv901S-"
   },
   "source": [
    "##### Get the integer representation of the newline character `ix_newline`\n",
    "* `ix_newline`: The newline character signals the end of the dinosaur name.\n",
    "    - get the integer representation of the newline character `'\\n'`.\n",
    "    - Use `char_to_ix`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KGnawmFe01S_"
   },
   "source": [
    "##### Set the list of labels (integer representation of the characters): `Y`\n",
    "* The goal is to train the RNN to predict the next letter in the name, so the labels are the list of characters that are one time step ahead of the characters in the input `X`.\n",
    "    - For example, `Y[0]` contains the same value as `X[1]`  \n",
    "* The RNN should predict a newline at the last letter so add ix_newline to the end of the labels. \n",
    "    - Append the integer representation of the newline character to the end of `Y`.\n",
    "    - Note that `append` is an in-place operation.\n",
    "    - It might be easier for you to add two lists together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "l214uNun01S_"
   },
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: model\n",
    "\n",
    "def model(data, ix_to_char, char_to_ix, num_iterations = 35000, n_a = 50, dino_names = 7, vocab_size = 27, verbose = False):\n",
    "    \"\"\"\n",
    "    Trains the model and generates dinosaur names. \n",
    "    \n",
    "    Arguments:\n",
    "    data -- text corpus\n",
    "    ix_to_char -- dictionary that maps the index to a character\n",
    "    char_to_ix -- dictionary that maps a character to an index\n",
    "    num_iterations -- number of iterations to train the model for\n",
    "    n_a -- number of units of the RNN cell\n",
    "    dino_names -- number of dinosaur names you want to sample at each iteration. \n",
    "    vocab_size -- number of unique characters found in the text (size of the vocabulary)\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- learned parameters\n",
    "    \"\"\"\n",
    "    \n",
    "    # Retrieve n_x and n_y from vocab_size\n",
    "    n_x, n_y = vocab_size, vocab_size\n",
    "    \n",
    "    # Initialize parameters\n",
    "    parameters = initialize_parameters(n_a, n_x, n_y)\n",
    "    \n",
    "    # Initialize loss (this is required because we want to smooth our loss, don't worry about it)\n",
    "    loss = get_initial_loss(vocab_size, dino_names)\n",
    "    \n",
    "    # Build list of all dinosaur names (training examples).\n",
    "    with open(\"dinos.txt\") as f:\n",
    "        examples = f.readlines()\n",
    "    examples = [x.lower().strip() for x in examples]\n",
    "    \n",
    "    # Shuffle list of all dinosaur names\n",
    "    np.random.seed(0)\n",
    "    np.random.shuffle(examples)\n",
    "    \n",
    "    # Initialize the hidden state of your LSTM\n",
    "    a_prev = np.zeros((n_a, 1))\n",
    "    \n",
    "    # Optimization loop\n",
    "    for j in range(num_iterations):\n",
    "        \n",
    "        ### START CODE HERE ###\n",
    "        \n",
    "        # Use the hint above to define one training example (X,Y) (≈ 2 lines)\n",
    "        index = j % len(examples)\n",
    "        X = [None] + [char_to_ix[ch] for ch in examples[index]]\n",
    "        Y = X[1:] + [char_to_ix['\\n']]\n",
    "        \n",
    "        # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters\n",
    "        # Choose a learning rate of 0.01\n",
    "        curr_loss, gradients, a_prev = optimize(X,Y,a_prev,parameters)        \n",
    "        ### END CODE HERE ###\n",
    "        \n",
    "        # Use a latency trick to keep the loss smooth. It happens here to accelerate the training.\n",
    "        loss = smooth(loss, curr_loss)\n",
    "\n",
    "        # Every 2000 Iteration, generate \"n\" characters thanks to sample() to check if the model is learning properly\n",
    "        if j % 2000 == 0:\n",
    "            \n",
    "            print('Iteration: %d, Loss: %f' % (j, loss) + '\\n')\n",
    "            \n",
    "            # The number of dinosaur names to print\n",
    "            seed = 0\n",
    "            for name in range(dino_names):\n",
    "                \n",
    "                # Sample indices and print them\n",
    "                sampled_indices = sample(parameters, char_to_ix, seed)\n",
    "                print_sample(sampled_indices, ix_to_char)\n",
    "                \n",
    "                seed += 1  # To get the same result for grading purposed, increment the seed by one. \n",
    "      \n",
    "            print('\\n')\n",
    "        \n",
    "    return parameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-HvBlAnS01TB"
   },
   "source": [
    "Run the following cell, you should observe your model outputting random-looking characters at the first iteration. After a few thousand iterations, your model should learn to generate reasonable-looking names. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3EH8Edc001TC",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 0, Loss: 23.087336\n",
      "\n",
      "Nkzxwtdmfqoeyhsqwasjkjvu\n",
      "Kneb\n",
      "Kzxwtdmfqoeyhsqwasjkjvu\n",
      "Neb\n",
      "Zxwtdmfqoeyhsqwasjkjvu\n",
      "Eb\n",
      "Xwtdmfqoeyhsqwasjkjvu\n",
      "\n",
      "\n",
      "Iteration: 2000, Loss: 27.884160\n",
      "\n",
      "Liusskeomnolxeros\n",
      "Hmdaairus\n",
      "Hytroligoraurus\n",
      "Lecalosapaus\n",
      "Xusicikoraurus\n",
      "Abalpsamantisaurus\n",
      "Tpraneronxeros\n",
      "\n",
      "\n",
      "Iteration: 4000, Loss: 25.901815\n",
      "\n",
      "Mivrosaurus\n",
      "Inee\n",
      "Ivtroplisaurus\n",
      "Mbaaisaurus\n",
      "Wusichisaurus\n",
      "Cabaselachus\n",
      "Toraperlethosdarenitochusthiamamumamaon\n",
      "\n",
      "\n",
      "Iteration: 6000, Loss: 24.608779\n",
      "\n",
      "Onwusceomosaurus\n",
      "Lieeaerosaurus\n",
      "Lxussaurus\n",
      "Oma\n",
      "Xusteonosaurus\n",
      "Eeahosaurus\n",
      "Toreonosaurus\n",
      "\n",
      "\n",
      "Iteration: 8000, Loss: 24.070350\n",
      "\n",
      "Onxusichepriuon\n",
      "Kilabersaurus\n",
      "Lutrodon\n",
      "Omaaerosaurus\n",
      "Xutrcheps\n",
      "Edaksoje\n",
      "Trodiktonus\n",
      "\n",
      "\n",
      "Iteration: 10000, Loss: 23.844446\n",
      "\n",
      "Onyusaurus\n",
      "Klecalosaurus\n",
      "Lustodon\n",
      "Ola\n",
      "Xusodonia\n",
      "Eeaeosaurus\n",
      "Troceosaurus\n",
      "\n",
      "\n",
      "Iteration: 12000, Loss: 23.291971\n",
      "\n",
      "Onyxosaurus\n",
      "Kica\n",
      "Lustrepiosaurus\n",
      "Olaagrraiansaurus\n",
      "Yuspangosaurus\n",
      "Eealosaurus\n",
      "Trognesaurus\n",
      "\n",
      "\n",
      "Iteration: 14000, Loss: 23.382338\n",
      "\n",
      "Meutromodromurus\n",
      "Inda\n",
      "Iutroinatorsaurus\n",
      "Maca\n",
      "Yusteratoptititan\n",
      "Ca\n",
      "Troclosaurus\n",
      "\n",
      "\n",
      "Iteration: 16000, Loss: 23.255630\n",
      "\n",
      "Meustolkanolus\n",
      "Indabestacarospceryradwalosaurus\n",
      "Justolopinaveraterasauracoptelalenyden\n",
      "Maca\n",
      "Yusocles\n",
      "Daahosaurus\n",
      "Trodon\n",
      "\n",
      "\n",
      "Iteration: 18000, Loss: 22.905483\n",
      "\n",
      "Phytronn\n",
      "Meicanstolanthus\n",
      "Mustrisaurus\n",
      "Pegalosaurus\n",
      "Yuskercis\n",
      "Egalosaurus\n",
      "Tromelosaurus\n",
      "\n",
      "\n",
      "Iteration: 20000, Loss: 22.873854\n",
      "\n",
      "Nlyushanerohyisaurus\n",
      "Loga\n",
      "Lustrhigosaurus\n",
      "Nedalosaurus\n",
      "Yuslangosaurus\n",
      "Elagosaurus\n",
      "Trrangosaurus\n",
      "\n",
      "\n",
      "Iteration: 22000, Loss: 22.710545\n",
      "\n",
      "Onyxromicoraurospareiosatrus\n",
      "Liga\n",
      "Mustoffankeugoptardoros\n",
      "Ola\n",
      "Yusodogongterosaurus\n",
      "Ehaerona\n",
      "Trododongxernochenhus\n",
      "\n",
      "\n",
      "Iteration: 24000, Loss: 22.604827\n",
      "\n",
      "Meustognathiterhucoplithaloptha\n",
      "Jigaadosaurus\n",
      "Kurrodon\n",
      "Mecaistheansaurus\n",
      "Yuromelosaurus\n",
      "Eiaeropeeton\n",
      "Troenathiteritaus\n",
      "\n",
      "\n",
      "Iteration: 26000, Loss: 22.714486\n",
      "\n",
      "Nhyxosaurus\n",
      "Kola\n",
      "Lvrosaurus\n",
      "Necalosaurus\n",
      "Yurolonlus\n",
      "Ejakosaurus\n",
      "Troindronykus\n",
      "\n",
      "\n",
      "Iteration: 28000, Loss: 22.647640\n",
      "\n",
      "Onyxosaurus\n",
      "Loceahosaurus\n",
      "Lustleonlonx\n",
      "Olabasicachudrakhurgawamosaurus\n",
      "Ytrojianiisaurus\n",
      "Eladon\n",
      "Tromacimathoshargicitan\n",
      "\n",
      "\n",
      "Iteration: 30000, Loss: 22.598485\n",
      "\n",
      "Oryuton\n",
      "Locaaesaurus\n",
      "Lustoendosaurus\n",
      "Olaahus\n",
      "Yusaurus\n",
      "Ehadopldarshuellus\n",
      "Troia\n",
      "\n",
      "\n",
      "Iteration: 32000, Loss: 22.211861\n",
      "\n",
      "Meutronlapsaurus\n",
      "Kracallthcaps\n",
      "Lustrathus\n",
      "Macairugeanosaurus\n",
      "Yusidoneraverataus\n",
      "Eialosaurus\n",
      "Troimaniathonsaurus\n",
      "\n",
      "\n",
      "Iteration: 34000, Loss: 22.447230\n",
      "\n",
      "Onyxipaledisons\n",
      "Kiabaeropa\n",
      "Lussiamang\n",
      "Pacaeptabalsaurus\n",
      "Xosalong\n",
      "Eiacoteg\n",
      "Troia\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "parameters = model(data, ix_to_char, char_to_ix, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QFv8SeTY01TE"
   },
   "source": [
    "** Expected Output**\n",
    "\n",
    "```Python\n",
    "j =  0 idx =  0\n",
    "single_example = turiasaurus\n",
    "single_example_chars ['t', 'u', 'r', 'i', 'a', 's', 'a', 'u', 'r', 'u', 's']\n",
    "single_example_ix [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19]\n",
    " X =  [None, 20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19] \n",
    " Y =        [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19, 0] \n",
    "\n",
    "Iteration: 0, Loss: 23.087336\n",
    "\n",
    "Nkzxwtdmfqoeyhsqwasjkjvu\n",
    "Kneb\n",
    "Kzxwtdmfqoeyhsqwasjkjvu\n",
    "Neb\n",
    "Zxwtdmfqoeyhsqwasjkjvu\n",
    "Eb\n",
    "Xwtdmfqoeyhsqwasjkjvu\n",
    "\n",
    "\n",
    "j =  1535 idx =  1535\n",
    "j =  1536 idx =  0\n",
    "Iteration: 2000, Loss: 27.884160\n",
    "\n",
    "...\n",
    "\n",
    "Iteration: 34000, Loss: 22.447230\n",
    "\n",
    "Onyxipaledisons\n",
    "Kiabaeropa\n",
    "Lussiamang\n",
    "Pacaeptabalsaurus\n",
    "Xosalong\n",
    "Eiacoteg\n",
    "Troia\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4mmYZwFu01TF"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "You can see that your algorithm has started to generate plausible dinosaur names towards the end of the training. At first, it was generating random characters, but towards the end you could see dinosaur names with cool endings. Feel free to run the algorithm even longer and play with hyperparameters to see if you can get even better results. Our implementation generated some really cool names like `maconucon`, `marloralus` and `macingsersaurus`. Your model hopefully also learned that dinosaur names tend to end in `saurus`, `don`, `aura`, `tor`, etc.\n",
    "\n",
    "If your model generates some non-cool names, don't blame the model entirely--not all actual dinosaur names sound cool. (For example, `dromaeosauroides` is an actual dinosaur name and is in the training set.) But this model should give you a set of candidates from which you can pick the coolest! \n",
    "\n",
    "This assignment had used a relatively small dataset, so that you could train an RNN quickly on a CPU. Training a model of the english language requires a much bigger dataset, and usually needs much more computation, and could run for many hours on GPUs. We ran our dinosaur name for quite some time, and so far our favorite name is the great, undefeatable, and fierce: Mangosaurus!\n",
    "\n",
    "<img src=\"images/mangosaurus.jpeg\" style=\"width:250;height:300px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2u55TY-W01TG"
   },
   "source": [
    "## 4 - Writing like Shakespeare\n",
    "\n",
    "The rest of this notebook is optional and is not graded, but we hope you'll do it anyway since it's quite fun and informative. \n",
    "\n",
    "A similar (but more complicated) task is to generate Shakespeare poems. Instead of learning from a dataset of Dinosaur names you can use a collection of Shakespearian poems. Using LSTM cells, you can learn longer term dependencies that span many characters in the text--e.g., where a character appearing somewhere a sequence can influence what should be a different character much much later in the sequence. These long term dependencies were less important with dinosaur names, since the names were quite short. \n",
    "\n",
    "\n",
    "<img src=\"images/shakespeare.jpg\" style=\"width:500;height:400px;\">\n",
    "<caption><center> Let's become poets! </center></caption>\n",
    "\n",
    "We have implemented a Shakespeare poem generator with Keras. Run the following cell to load the required packages and models. This may take a few minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dgPMYCZl01TH"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading text data...\n",
      "Creating training set...\n",
      "number of training examples: 31412\n",
      "Vectorizing training set...\n",
      "Loading model...\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from keras.callbacks import LambdaCallback\n",
    "from keras.models import Model, load_model, Sequential\n",
    "from keras.layers import Dense, Activation, Dropout, Input, Masking\n",
    "from keras.layers import LSTM\n",
    "from keras.utils.data_utils import get_file\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from shakespeare_utils import *\n",
    "import sys\n",
    "import io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jRaDcXrU01TJ"
   },
   "source": [
    "To save you some time, we have already trained a model for ~1000 epochs on a collection of Shakespearian poems called [*\"The Sonnets\"*](shakespeare.txt). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XGG0VdGv01TJ"
   },
   "source": [
    "Let's train the model for one more epoch. When it finishes training for an epoch---this will also take a few minutes---you can run `generate_output`, which will prompt asking you for an input (`<`40 characters). The poem will start with your sentence, and our RNN-Shakespeare will complete the rest of the poem for you! For example, try \"Forsooth this maketh no sense \" (don't enter the quotation marks). Depending on whether you include the space at the end, your results might also differ--try it both ways, and try other inputs as well. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y4DBjTlz01TK",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "31412/31412 [==============================] - 330s - loss: 2.5559   \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f3dfdea5978>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print_callback = LambdaCallback(on_epoch_end=on_epoch_end)\n",
    "\n",
    "model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wDoriy4Z01TM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write the beginning of your poem, the Shakespeare machine will complete it. Your input is: What a day!\n",
      "\n",
      "\n",
      "Here is your poem: \n",
      "\n",
      "What a day!\n",
      "therell bid or lameled, that from thus should) you, gaadey evernight,\n",
      "ang tire the commorved woor a do crued fates.\n",
      "i be thy rides to dadart denend,\n",
      "ame my nother's dis sancon it all eeced?\n",
      " \n",
      "so your sweet eid orfweth for my hi, to prucer, the thoudt weeld behind,\n",
      "when proad vo winl ant haa tire the befull.\n",
      "your ferline of then thing somner.\n",
      "so back of lrish of nowk hilf his trues,\n",
      "but by have do"
     ]
    }
   ],
   "source": [
    "# Run this cell to try with different inputs without having to re-train the model \n",
    "generate_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b1UA61-t01TO"
   },
   "source": [
    "The RNN-Shakespeare model is very similar to the one you have built for dinosaur names. The only major differences are:\n",
    "- LSTMs instead of the basic RNN to capture longer-range dependencies\n",
    "- The model is a deeper, stacked LSTM model (2 layer)\n",
    "- Using Keras instead of python to simplify the code \n",
    "\n",
    "If you want to learn more, you can also check out the Keras Team's text generation implementation on GitHub: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py.\n",
    "\n",
    "Congratulations on finishing this notebook! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2EgCyjte01TP"
   },
   "source": [
    "**References**:\n",
    "- This exercise took inspiration from Andrej Karpathy's implementation: https://gist.github.com/karpathy/d4dee566867f8291f086. To learn more about text generation, also check out Karpathy's [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n",
    "- For the Shakespearian poem generator, our implementation was based on the implementation of an LSTM text generator by the Keras team: https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "u9y2yxSI01TP"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "1dYg0",
   "launcher_item_id": "MLhxP"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}