LSTM Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating Text using an LSTM Network (No libraries)\n",
    "\n",
    "## Demo\n",
    "\n",
    "We'll train an LSTM network built in pure numpy to generate Eminem lyrics. LSTMs are a fairly simple extension to neural networks, and they're behind a lot of the amazing achievements deep learning has made in the past few years.\n",
    "\n",
    "## What is a Recurrent Network?\n",
    "\n",
    "Recurrent nets are cool, they're useful for learning sequences of data. Input. Hidden state. Output. \n",
    "![alt text](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png \"Logo Title Text 1\")\n",
    "\n",
    "It has a weight matrix that connects input to hidden state. But also a weight matrix that connects hidden state to hidden state at previous time step.\n",
    "![alt text](https://iamtrask.github.io/img/basic_recurrence_singleton.png \"Logo Title Text 1\")\n",
    "\n",
    "So we could even think of it as the same feedforward network connecting to itself overtime (unrolled) since passing in\n",
    "not just input in next training iteration but input + previous hidden state\n",
    "![alt text](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png \"Logo Title Text 1\")\n",
    "\n",
    "## The Problem with Recurrent Networks\n",
    "\n",
    "If we want to predict the last word in the sentence \"The grass is green\", that's totally doable. \n",
    "![alt text](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-shorttermdepdencies.png \"Logo Title Text 1\")\n",
    "\n",
    "But if we want to predict the last word in the sentence \"I am French (2000 words later) i speak fluent French\". We need to be able to remember long range dependencies. RNN's are bad at this. They forget the long term past easily.\n",
    "![alt text](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-longtermdependencies.png \"Logo Title Text 1\")\n",
    "\n",
    "This is called the \"Vanishing Gradient Problem\". The Gradient exponentially decays as its backpropagated\n",
    "![alt text](http://slideplayer.com/slide/5251503/16/images/6/Recurrent+Neural+Networks.jpg \"Logo Title Text 1\")\n",
    "![alt text](https://cdn-images-1.medium.com/max/1600/1*8JJ6sYleUtvUZR7TOyyFVg.png \"Logo Title Text 1\")\n",
    "\n",
    "There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.If either of these factors is smaller than 1, then the gradients may vanish in time; if larger than 1, then exploding might happen. \n",
    "\n",
    "But there exists a solution! Enter the LSTM Cell.\n",
    "\n",
    "## The LSTM Cell (Long-Short Term Memory Cell)\n",
    "\n",
    "We've placed no constraints on how our model updates, so its knowledge can change pretty chaotically: at one frame it thinks the characters are in the US, at the next frame it sees the characters eating sushi and thinks they're in Japan, and at the next frame it sees polar bears and thinks they're on Hydra Island. \n",
    "\n",
    "This chaos means information quickly transforms and vanishes, and it's difficult for the model to keep a long-term memory. So what you’d like is for the network to learn how to update its beliefs (scenes without Bob shouldn't change Bob-related information, scenes with Alice should focus on gathering details about her), in a way that its knowledge of the world evolves more gently.\n",
    "\n",
    "It replaces the normal RNN cell and uses an input, forget, and output gate. As well as a cell state\n",
    "![alt text](https://www.researchgate.net/profile/Mohsen_Fayyaz/publication/306377072/figure/fig2/AS:398082849165314@1471921755580/Fig-2-An-example-of-a-basic-LSTM-cell-left-and-a-basic-RNN-cell-right-Figure.ppm \"Logo Title Text 1\")\n",
    "\n",
    "![alt text](https://kijungyoon.github.io/assets/images/lstm.png \"Logo Title Text 1\")\n",
    "These gates each have their own set of weight values. The whole thing is differentiable (meaning we compute gradients and update the weights using them) so we can backprop through it\n",
    "\n",
    "We want our model to be able to know what to forget, what to remember. So when new a input comes in, the model first forgets any long-term information it decides it no longer needs. Then it learns which parts of the new input are worth using, and saves them into its long-term memory.\n",
    "\n",
    "And instead of using the full long-term memory all the time, it learns which parts to focus on instead.\n",
    "\n",
    "Basically, we need mechanisms for forgetting, remembering, and attention. That's what the LSTM cell provides us.\n",
    "\n",
    "Whereas a vanilla RNN uses one equation to update its hidden state/memory:\n",
    "![alt text](http://i.imgur.com/nT4VBPf.png \"Logo Title Text 1\")\n",
    "\n",
    "Which piece of long term memory to remember and forget? \n",
    "We'll use new input and working memory to learn remember gate. \n",
    "Which part of new data should we use and save? \n",
    "Update working memory using attention vector. \n",
    "\n",
    "- The long-term memory, is usually called the cell state, \n",
    "- The working memory, is usually called the hidden state. This is analogous to the hidden state in vanilla RNNs.\n",
    "- The remember vector, is usually called the forget gate (despite the fact that a 1 in the forget gate still means to keep the memory and a 0 still means to forget it),\n",
    "- The save vector, is usually called the input gate (as it determines how much of the input to let into the cell state), \n",
    "- The focus vector, is usually called the output gate )\n",
    "\n",
    "## Use cases\n",
    "\n",
    "\n",
    "Video \n",
    "[![IMAGE ALT TEXT HERE](http://img.youtube.com/vi/mLxsbWAYIpw/0.jpg)](https://www.youtube.com/watch?time_continue=14&v=mLxsbWAYIpw)\n",
    "\n",
    "The most popular application right now is actually in natural language processing\n",
    "which involves sequential data such as words,  sentences, sound spectrogram, etc. So applications with translation,  sentiment analysis,  text generation, etc. \n",
    "\n",
    "In other less obvious areas there's also applications of lstm.  Such as for image classification (feeding each picture's pixel in row by row). And even for deepmind's deep Q Learning agents.\n",
    "\n",
    "## Other great examples\n",
    "\n",
    "Speech recognition Tensorflow - https://github.com/zzw922cn/Automatic_Speech_Recognition\n",
    "LSTM visualization - https://github.com/HendrikStrobelt/LSTMVis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps \n",
    "\n",
    "1. Build RNN class\n",
    "2. Build LSTM Cell Class\n",
    "3. Data Loading Functions\n",
    "3. Training time!\n",
    "\n",
    "![alt text](http://eric-yuan.me/wp-content/uploads/2015/06/5.jpg \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class RecurrentNeuralNetwork:\n",
    "    #input (word), expected output (next word), num of words (num of recurrences), array expected outputs, learning rate\n",
    "    def __init__ (self, xs, ys, rl, eo, lr):\n",
    "        #initial input (first word)\n",
    "        self.x = np.zeros(xs)\n",
    "        #input size \n",
    "        self.xs = xs\n",
    "        #expected output (next word)\n",
    "        self.y = np.zeros(ys)\n",
    "        #output size\n",
    "        self.ys = ys\n",
    "        #weight matrix for interpreting results from LSTM cell (num words x num words matrix)\n",
    "        self.w = np.random.random((ys, ys))\n",
    "        #matrix used in RMSprop\n",
    "        self.G = np.zeros_like(self.w)\n",
    "        #length of the recurrent network - number of recurrences i.e num of words\n",
    "        self.rl = rl\n",
    "        #learning rate \n",
    "        self.lr = lr\n",
    "        #array for storing inputs\n",
    "        self.ia = np.zeros((rl+1,xs))\n",
    "        #array for storing cell states\n",
    "        self.ca = np.zeros((rl+1,ys))\n",
    "        #array for storing outputs\n",
    "        self.oa = np.zeros((rl+1,ys))\n",
    "        #array for storing hidden states\n",
    "        self.ha = np.zeros((rl+1,ys))\n",
    "        #forget gate \n",
    "        self.af = np.zeros((rl+1,ys))\n",
    "        #input gate\n",
    "        self.ai = np.zeros((rl+1,ys))\n",
    "        #cell state\n",
    "        self.ac = np.zeros((rl+1,ys))\n",
    "        #output gate\n",
    "        self.ao = np.zeros((rl+1,ys))\n",
    "        #array of expected output values\n",
    "        self.eo = np.vstack((np.zeros(eo.shape[0]), eo.T))\n",
    "        #declare LSTM cell (input, output, amount of recurrence, learning rate)\n",
    "        self.LSTM = LSTM(xs, ys, rl, lr)\n",
    "    \n",
    "    #activation function. simple nonlinearity, convert nums into probabilities between 0 and 1\n",
    "    def sigmoid(self, x):\n",
    "        return 1 / (1 + np.exp(-x))\n",
    "    \n",
    "    #the derivative of the sigmoid function. used to compute gradients for backpropagation\n",
    "    def dsigmoid(self, x):\n",
    "        return self.sigmoid(x) * (1 - self.sigmoid(x))    \n",
    "    \n",
    "    #lets apply a series of matrix operations to our input (curr word) to compute a predicted output (next word)\n",
    "    def forwardProp(self):\n",
    "        for i range(1, self.rl+1):\n",
    "            self.LSTM.x = np.hstack((self.ha[i-1], self.x))\n",
    "            cs, hs, f, c, o = self.LSTM.forwardProp()\n",
    "            #store computed cell state\n",
    "            self.ca[i] = cs\n",
    "            self.ha[i] = hs\n",
    "            self.af[i] = f\n",
    "            self.ai[i] = inp\n",
    "            self.ac[i] = c\n",
    "            self.ao[i] = o\n",
    "            self.oa[i] self.sigmoid(np.dot(self.w, hs))\n",
    "            self.x = self.eo[i-1]\n",
    "        return self.oa\n",
    "   \n",
    "    \n",
    "    def backProp(self):\n",
    "        #update our weight matrices (Both in our Recurrent network, as well as the weight matrices inside LSTM cell)\n",
    "        #init an empty error value \n",
    "        totalError = 0\n",
    "        #initialize matrices for gradient updates\n",
    "        #First, these are RNN level gradients\n",
    "        #cell state\n",
    "        dfcs = np.zeros(self.ys)\n",
    "        #hidden state,\n",
    "        dfhs = np.zeros(self.ys)\n",
    "        #weight matrix\n",
    "        tu = np.zeros((self.ys,self.ys))\n",
    "        #Next, these are LSTM level gradients\n",
    "        #forget gate\n",
    "        tfu = np.zeros((self.ys, self.xs+self.ys))\n",
    "        #input gate\n",
    "        tiu = np.zeros((self.ys, self.xs+self.ys))\n",
    "        #cell unit\n",
    "        tcu = np.zeros((self.ys, self.xs+self.ys))\n",
    "        #output gate\n",
    "        tou = np.zeros((self.ys, self.xs+self.ys))\n",
    "        #loop backwards through recurrences\n",
    "        for i in range(self.rl, -1, -1):\n",
    "            #error = calculatedOutput - expectedOutput\n",
    "            error = self.oa[i] - self.eo[i]\n",
    "            #calculate update for weight matrix\n",
    "            #(error * derivative of the output) * hidden state\n",
    "            tu += np.dot(np.atleast_2d(error * self.dsigmoid(self.oa[i])), np.atleast_2d(self.ha[i]).T)\n",
    "            #Time to propagate error back to exit of LSTM cell\n",
    "            #1. error * RNN weight matrix\n",
    "            error = np.dot(error, self.w)\n",
    "            #2. set input values of LSTM cell for recurrence i (horizontal stack of arrays, hidden + input)\n",
    "            self.LSTM.x = np.hstack((self.ha[i-1], self.ia[i]))\n",
    "            #3. set cell state of LSTM cell for recurrence i (pre-updates)\n",
    "            self.LSTM.cs = self.ca[i]\n",
    "            #Finally, call the LSTM cell's backprop, retreive gradient updates\n",
    "            #gradient updates for forget, input, cell unit, and output gates + cell states & hiddens states\n",
    "            fu, iu, cu, ou, dfcs, dfhs = self.LSTM.backProp(error, self.ca[i-1], self.af[i], self.ai[i], self.ac[i], self.ao[i], dfcs, dfhs)\n",
    "            #calculate total error (not necesarry, used to measure training progress)\n",
    "            totalError += np.sum(error)\n",
    "            #accumulate all gradient updates\n",
    "            #forget gate\n",
    "            tfu += fu\n",
    "            #input gate\n",
    "            tiu += iu\n",
    "            #cell state\n",
    "            tcu += cu\n",
    "            #output gate\n",
    "            tou += ou\n",
    "        #update LSTM matrices with average of accumulated gradient updates    \n",
    "        self.LSTM.update(tfu/self.rl, tiu/self.rl, tcu/self.rl, tou/self.rl) \n",
    "        #update weight matrix with average of accumulated gradient updates  \n",
    "        self.update(tu/self.rl)\n",
    "        #return total error of this iteration\n",
    "        return totalError\n",
    "    \n",
    "    def update(self, u):\n",
    "        #vanilla implementation of RMSprop\n",
    "        self.G = 0.9 * self.G + 0.1 * u**2  \n",
    "        self.w -= self.lr/np.sqrt(self.G + 1e-8) * u\n",
    "        return\n",
    "    \n",
    "    #this is where we generate some sample text after having fully trained our model\n",
    "    #i.e error is below some threshold\n",
    "    def sample(self):\n",
    "         #loop through recurrences - start at 1 so the 0th entry of all arrays will be an array of 0's\n",
    "        for i in range(1, self.rl+1):\n",
    "            #set input for LSTM cell, combination of input (previous output) and previous hidden state\n",
    "            self.LSTM.x = np.hstack((self.ha[i-1], self.x))\n",
    "            #run forward prop on the LSTM cell, retrieve cell state and hidden state\n",
    "            cs, hs, f, inp, c, o = self.LSTM.forwardProp()\n",
    "            #store input as vector\n",
    "            maxI = np.argmax(self.x)\n",
    "            self.x = np.zeros_like(self.x)\n",
    "            self.x[maxI] = 1\n",
    "            self.ia[i] = self.x #Use np.argmax?\n",
    "            #store cell states\n",
    "            self.ca[i] = cs\n",
    "            #store hidden state\n",
    "            self.ha[i] = hs\n",
    "            #forget gate\n",
    "            self.af[i] = f\n",
    "            #input gate\n",
    "            self.ai[i] = inp\n",
    "            #cell state\n",
    "            self.ac[i] = c\n",
    "            #output gate\n",
    "            self.ao[i] = o\n",
    "            #calculate output by multiplying hidden state with weight matrix\n",
    "            self.oa[i] = self.sigmoid(np.dot(self.w, hs))\n",
    "            #compute new input\n",
    "            maxI = np.argmax(self.oa[i])\n",
    "            newX = np.zeros_like(self.x)\n",
    "            newX[maxI] = 1\n",
    "            self.x = newX\n",
    "        #return all outputs    \n",
    "        return self.oa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](http://i.imgur.com/BUAVEZg.png \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LSTM:\n",
    "    # LSTM cell (input, output, amount of recurrence, learning rate)\n",
    "    def __init__ (self, xs, ys, rl, lr):\n",
    "        #input is word length x word length\n",
    "        self.x = np.zeros(xs+ys)\n",
    "        #input size is word length + word length\n",
    "        self.xs = xs + ys\n",
    "        #output \n",
    "        self.y = np.zeros(ys)\n",
    "        #output size\n",
    "        self.ys = ys\n",
    "        #cell state intialized as size of prediction\n",
    "        self.cs = np.zeros(ys)\n",
    "        #how often to perform recurrence\n",
    "        self.rl = rl\n",
    "        #balance the rate of training (learning rate)\n",
    "        self.lr = lr\n",
    "        #init weight matrices for our gates\n",
    "        #forget gate\n",
    "        self.f = np.random.random((ys, xs+ys))\n",
    "        #input gate\n",
    "        self.i = np.random.random((ys, xs+ys))\n",
    "        #cell state\n",
    "        self.c = np.random.random((ys, xs+ys))\n",
    "        #output gate\n",
    "        self.o = np.random.random((ys, xs+ys))\n",
    "        #forget gate gradient\n",
    "        self.Gf = np.zeros_like(self.f)\n",
    "        #input gate gradient\n",
    "        self.Gi = np.zeros_like(self.i)\n",
    "        #cell state gradient\n",
    "        self.Gc = np.zeros_like(self.c)\n",
    "        #output gate gradient\n",
    "        self.Go = np.zeros_like(self.o)\n",
    "    \n",
    "    #activation function to activate our forward prop, just like in any type of neural network\n",
    "    def sigmoid(self, x):\n",
    "        return 1 / (1 + np.exp(-x))\n",
    "    \n",
    "    #derivative of sigmoid to help computes gradients\n",
    "    def dsigmoid(self, x):\n",
    "        return self.sigmoid(x) * (1 - self.sigmoid(x))\n",
    "    \n",
    "    #tanh! another activation function, often used in LSTM cells\n",
    "    #Having stronger gradients: since data is centered around 0, \n",
    "    #the derivatives are higher. To see this, calculate the derivative \n",
    "    #of the tanh function and notice that input values are in the range [0,1].\n",
    "    def tangent(self, x):\n",
    "        return np.tanh(x)\n",
    "    \n",
    "    #derivative for computing gradients\n",
    "    def dtangent(self, x):\n",
    "        return 1 - np.tanh(x)**2\n",
    "    \n",
    "    #lets compute a series of matrix multiplications to convert our input into our output\n",
    "    def forwardProp(self):\n",
    "        f = self.sigmoid(np.dot(self.f, self.x))\n",
    "        self.cs *= f\n",
    "        i = self.sigmoid(np.dot(self.i, self.x))\n",
    "        c = self.tangent(np.dot(self.c, self.x))\n",
    "        self.cs += i * c\n",
    "        o = self.sigmoid(np.dot(self.o, self.x))\n",
    "        self.y = o * self.tangent(self.cs)\n",
    "        return self.cs, self.y, f, i, c, o\n",
    "    \n",
    "   \n",
    "    def backProp(self, e, pcs, f, i, c, o, dfcs, dfhs):\n",
    "        #error = error + hidden state derivative. clip the value between -6 and 6.\n",
    "        e = np.clip(e + dfhs, -6, 6)\n",
    "        #multiply error by activated cell state to compute output derivative\n",
    "        do = self.tangent(self.cs) * e\n",
    "        #output update = (output deriv * activated output) * input\n",
    "        ou = np.dot(np.atleast_2d(do * self.dtangent(o)).T, np.atleast_2d(self.x))\n",
    "        #derivative of cell state = error * output * deriv of cell state + deriv cell\n",
    "        dcs = np.clip(e * o * self.dtangent(self.cs) + dfcs, -6, 6)\n",
    "        #deriv of cell = deriv cell state * input\n",
    "        dc = dcs * i\n",
    "        #cell update = deriv cell * activated cell * input\n",
    "        cu = np.dot(np.atleast_2d(dc * self.dtangent(c)).T, np.atleast_2d(self.x))\n",
    "        #deriv of input = deriv cell state * cell\n",
    "        di = dcs * c\n",
    "        #input update = (deriv input * activated input) * input\n",
    "        iu = np.dot(np.atleast_2d(di * self.dsigmoid(i)).T, np.atleast_2d(self.x))\n",
    "        #deriv forget = deriv cell state * all cell states\n",
    "        df = dcs * pcs\n",
    "        #forget update = (deriv forget * deriv forget) * input\n",
    "        fu = np.dot(np.atleast_2d(df * self.dsigmoid(f)).T, np.atleast_2d(self.x))\n",
    "        #deriv cell state = deriv cell state * forget\n",
    "        dpcs = dcs * f\n",
    "        #deriv hidden state = (deriv cell * cell) * output + deriv output * output * output deriv input * input * output + deriv forget\n",
    "        #* forget * output\n",
    "        dphs = np.dot(dc, self.c)[:self.ys] + np.dot(do, self.o)[:self.ys] + np.dot(di, self.i)[:self.ys] + np.dot(df, self.f)[:self.ys] \n",
    "        #return update gradinets for forget, input, cell, output, cell state, hidden state\n",
    "        return fu, iu, cu, ou, dpcs, dphs\n",
    "            \n",
    "    def update(self, fu, iu, cu, ou):\n",
    "        #update forget, input, cell, and output gradients\n",
    "        self.Gf = 0.9 * self.Gf + 0.1 * fu**2 \n",
    "        self.Gi = 0.9 * self.Gi + 0.1 * iu**2   \n",
    "        self.Gc = 0.9 * self.Gc + 0.1 * cu**2   \n",
    "        self.Go = 0.9 * self.Go + 0.1 * ou**2   \n",
    "        \n",
    "        #update our gates using our gradients\n",
    "        self.f -= self.lr/np.sqrt(self.Gf + 1e-8) * fu\n",
    "        self.i -= self.lr/np.sqrt(self.Gi + 1e-8) * iu\n",
    "        self.c -= self.lr/np.sqrt(self.Gc + 1e-8) * cu\n",
    "        self.o -= self.lr/np.sqrt(self.Go + 1e-8) * ou\n",
    "        return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def LoadText():\n",
    "    #open text and return input and output data (series of words)\n",
    "    with open(\"eminem.txt\", \"r\") as text_file:\n",
    "        data = text_file.read()\n",
    "    text = list(data)\n",
    "    outputSize = len(text)\n",
    "    data = list(set(text))\n",
    "    uniqueWords, dataSize = len(data), len(data) \n",
    "    returnData = np.zeros((uniqueWords, dataSize))\n",
    "    for i in range(0, dataSize):\n",
    "        returnData[i][i] = 1\n",
    "    returnData = np.append(returnData, np.atleast_2d(data), axis=0)\n",
    "    output = np.zeros((uniqueWords, outputSize))\n",
    "    for i in range(0, outputSize):\n",
    "        index = np.where(np.asarray(data) == text[i])\n",
    "        output[:,i] = returnData[0:-1,index[0]].astype(float).ravel()  \n",
    "    return returnData, uniqueWords, output, outputSize, data\n",
    "\n",
    "#write the predicted output (series of words) to disk\n",
    "def ExportText(output, data):\n",
    "    finalOutput = np.zeros_like(output)\n",
    "    prob = np.zeros_like(output[0])\n",
    "    outputText = \"\"\n",
    "    print(len(data))\n",
    "    print(output.shape[0])\n",
    "    for i in range(0, output.shape[0]):\n",
    "        for j in range(0, output.shape[1]):\n",
    "            prob[j] = output[i][j] / np.sum(output[i])\n",
    "        outputText += np.random.choice(data, p=prob)    \n",
    "    with open(\"output.txt\", \"w\") as text_file:\n",
    "        text_file.write(outputText)\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beginning\n",
      "Done Reading\n"
     ]
    }
   ],
   "source": [
    "#Begin program    \n",
    "print(\"Beginning\")\n",
    "iterations = 5000\n",
    "learningRate = 0.001\n",
    "#load input output data (words)\n",
    "returnData, numCategories, expectedOutput, outputSize, data = LoadText()\n",
    "print(\"Done Reading\")\n",
    "#init our RNN using our hyperparams and dataset\n",
    "RNN = RecurrentNeuralNetwork(numCategories, numCategories, outputSize, expectedOutput, learningRate)\n",
    "\n",
    "#training time!\n",
    "for i in range(1, iterations):\n",
    "    #compute predicted next word\n",
    "    RNN.forwardProp()\n",
    "    #update all our weights using our error\n",
    "    error = RNN.backProp()\n",
    "    #once our error/loss is small enough\n",
    "    print(\"Error on iteration \", i, \": \", error)\n",
    "    if error > -100 and error < 100 or i % 100 == 0:\n",
    "        #we can finally define a seed word\n",
    "        seed = np.zeros_like(RNN.x)\n",
    "        maxI = np.argmax(np.random.random(RNN.x.shape))\n",
    "        seed[maxI] = 1\n",
    "        RNN.x = seed  \n",
    "        #and predict some new text!\n",
    "        output = RNN.sample()\n",
    "        print(output)    \n",
    "        #write it all to disk\n",
    "        ExportText(output, data)\n",
    "        print(\"Done Writing\")\n",
    "print(\"Complete\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}