3.2 RNN and LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent Neural networks\n",
    "====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src=\"imgs/rnn.png\" width=\"20%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "keras.layers.recurrent.SimpleRNN(units, activation='tanh', use_bias=True, \n",
    "                                 kernel_initializer='glorot_uniform', \n",
    "                                 recurrent_initializer='orthogonal', \n",
    "                                 bias_initializer='zeros', \n",
    "                                 kernel_regularizer=None, \n",
    "                                 recurrent_regularizer=None, \n",
    "                                 bias_regularizer=None, \n",
    "                                 activity_regularizer=None, \n",
    "                                 kernel_constraint=None, recurrent_constraint=None, \n",
    "                                 bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arguments:\n",
    "\n",
    "<ul>\n",
    "<li><strong>units</strong>: Positive integer, dimensionality of the output space.</li>\n",
    "<li><strong>activation</strong>: Activation function to use\n",
    "    (see <a href=\"http://keras.io/activations/\">activations</a>).\n",
    "    If you pass None, no activation is applied\n",
    "    (ie. \"linear\" activation: <code>a(x) = x</code>).</li>\n",
    "<li><strong>use_bias</strong>: Boolean, whether the layer uses a bias vector.</li>\n",
    "<li><strong>kernel_initializer</strong>: Initializer for the <code>kernel</code> weights matrix,\n",
    "    used for the linear transformation of the inputs.\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>recurrent_initializer</strong>: Initializer for the <code>recurrent_kernel</code>\n",
    "    weights matrix,\n",
    "    used for the linear transformation of the recurrent state.\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>bias_initializer</strong>: Initializer for the bias vector\n",
    "    (see <a href=\"https://keras.io/initializers/\">initializers</a>).</li>\n",
    "<li><strong>kernel_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>recurrent_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>recurrent_kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>bias_regularizer</strong>: Regularizer function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>activity_regularizer</strong>: Regularizer function applied to\n",
    "    the output of the layer (its \"activation\").\n",
    "    (see <a href=\"https://keras.io/regularizers/\">regularizer</a>).</li>\n",
    "<li><strong>kernel_constraint</strong>: Constraint function applied to\n",
    "    the <code>kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "<li><strong>recurrent_constraint</strong>: Constraint function applied to\n",
    "    the <code>recurrent_kernel</code> weights matrix\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "<li><strong>bias_constraint</strong>: Constraint function applied to the bias vector\n",
    "    (see <a href=\"https://keras.io/constraints/\">constraints</a>).</li>\n",
    "<li><strong>dropout</strong>: Float between 0 and 1.\n",
    "    Fraction of the units to drop for\n",
    "    the linear transformation of the inputs.</li>\n",
    "<li><strong>recurrent_dropout</strong>: Float between 0 and 1.\n",
    "    Fraction of the units to drop for\n",
    "    the linear transformation of the recurrent state.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Backprop Through time  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrary to feed-forward neural networks, the RNN is characterized by the ability of encoding longer past information, thus very suitable for sequential models. The BPTT extends the ordinary BP algorithm to suit the recurrent neural\n",
    "architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "source": [
    "<img src=\"imgs/rnn2.png\" width=\"45%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reference**: [Backpropagation through Time](http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/user/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using Theano backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "import keras \n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "# -- Keras Import\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.preprocessing import image\n",
    "\n",
    "from keras.datasets import imdb\n",
    "from keras.datasets import mnist\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "\n",
    "from keras.utils import np_utils\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.recurrent import LSTM, GRU, SimpleRNN\n",
    "\n",
    "from keras.layers import Activation, TimeDistributed, RepeatVector\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### IMDB sentiment classification task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. \n",
    "\n",
    "IMDB provided a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. \n",
    "\n",
    "There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. \n",
    "\n",
    "http://ai.stanford.edu/~amaas/data/sentiment/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Preparation - IMDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 22s 1us/step\n",
      "17473536/17464789 [==============================] - 22s 1us/step\n",
      "(25000, 'train sequences')\n",
      "(25000, 'test sequences')\n",
      "Example:\n",
      "[list([1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32])]\n",
      "Pad sequences (samples x time)\n",
      "('X_train shape:', (25000, 100))\n",
      "('X_test shape:', (25000, 100))\n"
     ]
    }
   ],
   "source": [
    "max_features = 20000\n",
    "maxlen = 100  # cut texts after this number of words (among top max_features most common words)\n",
    "batch_size = 32\n",
    "\n",
    "print(\"Loading data...\")\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(X_train), 'train sequences')\n",
    "print(len(X_test), 'test sequences')\n",
    "\n",
    "print('Example:')\n",
    "print(X_train[:1])\n",
    "\n",
    "print(\"Pad sequences (samples x time)\")\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('X_test shape:', X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model building "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n",
      "Train...\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/1\n",
      "25000/25000 [==============================] - 156s 6ms/step - loss: 0.7212 - val_loss: 0.6781\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd9846a0d50>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128, input_length=maxlen))\n",
    "model.add(SimpleRNN(128))  \n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam')\n",
    "\n",
    "print(\"Train...\")\n",
    "model.fit(X_train, y_train, batch_size=batch_size, epochs=1, \n",
    "          validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSTM  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A LSTM network is an artificial neural network that contains LSTM blocks instead of, or in addition to, regular network units. A LSTM block may be described as a \"smart\" network unit that can remember a value for an arbitrary length of time. \n",
    "\n",
    "Unlike traditional RNNs, an Long short-term memory network is well-suited to learn from experience to classify, process and predict time series when there are very long time lags of unknown size between important events."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "source": [
    "<img src=\"imgs/gru.png\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "keras.layers.recurrent.LSTM(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, \n",
    "                            kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', \n",
    "                            bias_initializer='zeros', unit_forget_bias=True, kernel_regularizer=None, \n",
    "                            recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, \n",
    "                            kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, \n",
    "                            dropout=0.0, recurrent_dropout=0.0)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arguments\n",
    "\n",
    "<ul>\n",
    "<li><strong>units</strong>: Positive integer, dimensionality of the output space.</li>\n",
    "<li><strong>activation</strong>: Activation function to use\n",
    "    If you pass None, no activation is applied\n",
    "    (ie. \"linear\" activation: <code>a(x) = x</code>).</li>\n",
    "<li><strong>recurrent_activation</strong>: Activation function to use\n",
    "    for the recurrent step.</li>\n",
    "<li><strong>use_bias</strong>: Boolean, whether the layer uses a bias vector.</li>\n",
    "<li><strong>kernel_initializer</strong>: Initializer for the <code>kernel</code> weights matrix,\n",
    "    used for the linear transformation of the inputs.</li>\n",
    "<li><strong>recurrent_initializer</strong>: Initializer for the <code>recurrent_kernel</code>\n",
    "    weights matrix,\n",
    "    used for the linear transformation of the recurrent state.</li>\n",
    "<li><strong>bias_initializer</strong>: Initializer for the bias vector.</li>\n",
    "<li><strong>unit_forget_bias</strong>: Boolean.\n",
    "    If True, add 1 to the bias of the forget gate at initialization.\n",
    "    Setting it to true will also force <code>bias_initializer=\"zeros\"</code>.\n",
    "    This is recommended in <a href=\"http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf\">Jozefowicz et al.</a></li>\n",
    "<li><strong>kernel_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>kernel</code> weights matrix.</li>\n",
    "<li><strong>recurrent_regularizer</strong>: Regularizer function applied to\n",
    "    the <code>recurrent_kernel</code> weights matrix.</li>\n",
    "<li><strong>bias_regularizer</strong>: Regularizer function applied to the bias vector.</li>\n",
    "<li><strong>activity_regularizer</strong>: Regularizer function applied to\n",
    "    the output of the layer (its \"activation\").</li>\n",
    "<li><strong>kernel_constraint</strong>: Constraint function applied to\n",
    "    the <code>kernel</code> weights matrix.</li>\n",
    "<li><strong>recurrent_constraint</strong>: Constraint function applied to\n",
    "    the <code>recurrent_kernel</code> weights matrix.</li>\n",
    "<li><strong>bias_constraint</strong>: Constraint function applied to the bias vector.</li>\n",
    "<li><strong>dropout</strong>: Float between 0 and 1.\n",
    "    Fraction of the units to drop for\n",
    "    the linear transformation of the inputs.</li>\n",
    "<li><strong>recurrent_dropout</strong>: Float between 0 and 1.\n",
    "    Fraction of the units to drop for\n",
    "    the linear transformation of the recurrent state.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GRU  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gated recurrent units are a gating mechanism in recurrent neural networks. \n",
    "\n",
    "Much similar to the LSTMs, they have fewer parameters than LSTM, as they lack an output gate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "keras.layers.recurrent.GRU(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, \n",
    "                           kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', \n",
    "                           bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, \n",
    "                           bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, \n",
    "                           recurrent_constraint=None, bias_constraint=None, \n",
    "                           dropout=0.0, recurrent_dropout=0.0)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your Turn! - Hands on Rnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n",
      "Train...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO (theano.gof.compilelock): Waiting for existing lock by process '4617' (I am process '5021')\n",
      "INFO (theano.gof.compilelock): To manually release the lock, delete /home/user/.theano/compiledir_Linux-4.8--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.12-64/lock_dir\n",
      "INFO (theano.gof.compilelock): Waiting for existing lock by process '4617' (I am process '5021')\n",
      "INFO (theano.gof.compilelock): To manually release the lock, delete /home/user/.theano/compiledir_Linux-4.8--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.12-64/lock_dir\n"
     ]
    }
   ],
   "source": [
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128, input_length=maxlen))\n",
    "\n",
    "# !!! Play with those! try and get better results!\n",
    "#model.add(SimpleRNN(128))  \n",
    "#model.add(GRU(128))  \n",
    "#model.add(LSTM(128))  \n",
    "\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam')\n",
    "\n",
    "print(\"Train...\")\n",
    "model.fit(X_train, y_train, batch_size=batch_size, \n",
    "          epochs=4, validation_data=(X_test, y_test))\n",
    "score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence Generation using RNN(LSTM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = get_file('nietzsche.txt', origin=\"https://s3.amazonaws.com/text-datasets/nietzsche.txt\")\n",
    "#path = 'YourFile.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/user/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using Theano backend.\n",
      "WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:\n",
      " https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29\n",
      "\n",
      "Using gpu device 0: GeForce GTX 1050 Ti (CNMeM is enabled with initial size: 3.0% of memory, cuDNN 5105)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/nietzsche.txt\n",
      "606208/600901 [==============================] - 1s 2us/step\n",
      "614400/600901 [==============================] - 1s 2us/step\n",
      "('corpus length:', 600901)\n",
      "('total chars:', 59)\n",
      "('nb sequences:', 200287)\n",
      "Vectorization...\n",
      "Build model...\n",
      "()\n",
      "--------------------------------------------------\n",
      "('Iteration', 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/user/anaconda2/lib/python2.7/site-packages/keras/models.py:981: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  warnings.warn('The `nb_epoch` argument in `fit` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "200287/200287 [==============================] - 44s 219us/step - loss: 2.0034\n",
      "()\n",
      "('----- diversity:', 0.2)\n",
      "----- Generating with seed: \"--it is supposed to be a recommendation \"\n",
      "--it is supposed to be a recommendation of the soul the sould the self-decience of the thing the sould the something in the that the himself the the and the consequently the most the sense of the serman of the to the man the to the soul the something the soul to the man to the man for the man the sould the something and the to the serman the something to the to the are the the consequently the something the something the sould to the pr()\n",
      "()\n",
      "('----- diversity:', 0.5)\n",
      "----- Generating with seed: \"--it is supposed to be a recommendation \"\n",
      "--it is supposed to be a recommendation of the\n",
      "moral to dequire and it is to himself, the oppossitions of the hand superien of and who speet the are intistions being the of to the cares and to strenger of the accouses and the sime to the grouse the disprice of the manyer to disintery and the laush of the the perhaps the graws the reast and which the his of the consect to the of the and the tare the\n",
      "words something and to the consuch for()\n",
      "()\n",
      "('----- diversity:', 1.0)\n",
      "----- Generating with seed: \"--it is supposed to be a recommendation \"\n",
      "--it is supposed to be a recommendation hato as chra\n",
      "case endial cerfifull and tonfile the to caul for sublepous  to the good strong day to supiting pertanerating to greause may havisely not raies of rurtacity--meass the than? that to sechire, as but ben bithturally the entory impligentiousing fingualy its est, to to fallate, \"for, to arivire for indusess of\n",
      "eistos\" of theid must of to to genacne agneal not as not a bots, for the sereff()\n",
      "()\n",
      "('----- diversity:', 1.2)\n",
      "----- Generating with seed: \"--it is supposed to be a recommendation \"\n",
      "--it is supposed to be a recommendation of inruttle suffe why, wolld than\n",
      "the from and depails, tafl butry cerpa, not man, new\n",
      "ir tydar, they of-towing evenconachectly that poontome what strities moroluudteraled been, of\n",
      "aund beilund\n",
      "aradpe conception, sameanofatirinaciul eredicm,\"\"--it us--what \"in whincatly wenlu, demigo. to heasudpe experiams dooms and sposuloup. a wish--whed is perieovable athind,\n",
      "how clelilize,\n",
      "from his upilies, no()\n",
      "()\n",
      "--------------------------------------------------\n",
      "('Iteration', 2)\n",
      "Epoch 1/1\n",
      " 23680/200287 [==>...........................] - ETA: 39s - loss: 1.6761"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout\n",
    "from keras.layers import LSTM\n",
    "from keras.optimizers import RMSprop\n",
    "from keras.utils.data_utils import get_file\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import sys\n",
    "\n",
    "text = open(path).read().lower()\n",
    "print('corpus length:', len(text))\n",
    "\n",
    "chars = sorted(list(set(text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "\n",
    "# cut the text in semi-redundant sequences of maxlen characters\n",
    "maxlen = 40\n",
    "step = 3\n",
    "sentences = []\n",
    "next_chars = []\n",
    "for i in range(0, len(text) - maxlen, step):\n",
    "    sentences.append(text[i: i + maxlen])\n",
    "    next_chars.append(text[i + maxlen])\n",
    "print('nb sequences:', len(sentences))\n",
    "\n",
    "print('Vectorization...')\n",
    "X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        X[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1\n",
    "\n",
    "\n",
    "# build the model: a single LSTM\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n",
    "model.add(Dense(len(chars)))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)\n",
    "\n",
    "\n",
    "def sample(preds, temperature=1.0):\n",
    "    # helper function to sample an index from a probability array\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)\n",
    "\n",
    "# train the model, output generated text after each iteration\n",
    "for iteration in range(1, 60):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(X, y, batch_size=128, nb_epoch=1)\n",
    "\n",
    "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
    "\n",
    "    for diversity in [0.2, 0.5, 1.0, 1.2]:\n",
    "        print()\n",
    "        print('----- diversity:', diversity)\n",
    "\n",
    "        generated = ''\n",
    "        sentence = text[start_index: start_index + maxlen]\n",
    "        generated += sentence\n",
    "        print('----- Generating with seed: \"' + sentence + '\"')\n",
    "        sys.stdout.write(generated)\n",
    "\n",
    "        for i in range(400):\n",
    "            x = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(sentence):\n",
    "                x[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(x, verbose=0)[0]\n",
    "            next_index = sample(preds, diversity)\n",
    "            next_char = indices_char[next_index]\n",
    "\n",
    "            generated += next_char\n",
    "            sentence = sentence[1:] + next_char\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}