recurrently-happy-rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Having Fun with Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**~$ whoami**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Sergio G. Burdisso (sergio.burdisso@gmail.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**~$ pwd**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Universidad Nacional de San Luis, Argentina"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Having Fun with Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**~$ cat recommended_readings.txt**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "\n",
    "http://colah.github.io/posts/2015-08-Understanding-LSTMs/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Artificial Neural Networks\n",
    "### Classical vs. Recurrent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/0/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# RNN Unfolding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/0/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# RNN Stacking\n",
    "### Going deeper!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/0/4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Vanilla RNN\n",
    "### (diagram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/0/5.png)\n",
    "_(image taken from [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/))_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Vanilla RNN\n",
    "### (math)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**INPUT:** $X = X_t\\Vert H_{t-1}$\n",
    "\n",
    "(Where the $\\Vert$ operator denotes _concatenation_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**OUTPUT - NEXT HIDDEN STATE:**  $H_t = tanh(X\\cdot W_H + b_H)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**OPTIONAL OUTPUT - CLASSIFICATION:**  $Y_t = \\sigma(H_t\\cdot W + b)$\n",
    "\n",
    "(where $\\sigma$ denotes the $softmax$ function)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Long Short-Term Memory (LSTM) Network _[[paper](http://www.bioinf.jku.at/publications/older/2604.pdf)]_\n",
    "### (diagram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/0/6.png)\n",
    "_(image taken from [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/))_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Long Short-Term Memory (LSTM) Network\n",
    "### (math)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**INPUT:**<br>\n",
    "$X = X_t\\Vert H_{t-1}$<br>\n",
    "$X' = tanh(X\\cdot W_c + b_c)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**FORGET GATE:** $f = \\sigma(X\\cdot W_f + b_f)$<br>\n",
    "**UPDATE GATE:** $u = \\sigma(X\\cdot W_u + b_u)$<br>\n",
    "**RESULT GATE:** $r = \\sigma(X\\cdot W_r + b_r)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**LONG-SHORT MEMORY STATE:** $C_t = f\\odot C_{t-1} + u\\odot X'$<br>\n",
    "_(Where $\\odot$ denotes element-wise multiplication)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**OUTPUT - NEXT HIDDEN STATE:**  $H_t = r\\odot tanh(C_t)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Gated Recurrent Unit (GRU) _[[paper](http://arxiv.org/pdf/1406.1078v3.pdf)]_\n",
    "### (GRU is simpler than standard LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**INPUT:**<br>\n",
    "$X = X_t\\Vert H_{t-1}$<br>\n",
    "$X' = X_t\\Vert (r\\times H_{t-1})$<br>\n",
    "$X'' = tanh(X'\\cdot W_c + b_c)$<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**FORGET AND UPDATE GATE:** $z = \\sigma(X\\cdot W_z + b_z)$<br>\n",
    "**RESULT GATE:** $r = \\sigma(X\\cdot W_r + b_r)$<br>\n",
    "_(Note there's 2 Gates instead of 3 = fewer weights!)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**OUTPUT - NEXT HIDDEN STATE:**  $H_t = (1-z)\\odot H_{t-1} + z\\odot X''$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TASK: Creating our own RNN to generate sequence of characters (using TensorFlow)\n",
    "\n",
    "i.e. Our classifier needs to learn to predict the next character in a sequence!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "from tensorflow.contrib import layers\n",
    "from tensorflow.contrib import rnn\n",
    "\n",
    "from idataset import Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "hStateSize = 512  # Number of Hidden Units (NHU) i.e. size of Hidden State Vector\n",
    "maxSeqLength = 128  # MSL\n",
    "nLayers = 3  # NL\n",
    "\n",
    "learningRate = 1e-3  # 0.001\n",
    "dropoutProb = 0.3\n",
    "\n",
    "batchSize = 200  # BS\n",
    "alphaSize = Dataset.get_alphabet_size()  # AS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "hStateSize = 512   # Number of Hidden Units (NHU) i.e. Size of Hidden State Vector\n",
    "alphaSize = Dataset.get_alphabet_size(); # AS\n",
    "```\n",
    "![](./images/1/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "nLayers = 3  # NL\n",
    "```\n",
    "![](./images/1/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "maxSeqLength = 128  # MSL\n",
    "```\n",
    "![](./images/1/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "batchSize = 200  # BS\n",
    "```\n",
    "![](./images/1/4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dropout (=0)\n",
    "```python\n",
    "dropoutProb = 0.3\n",
    "```\n",
    "![](./images/1/5-0.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dropout (> 0)\n",
    "```python\n",
    "dropoutProb = 0.3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/1/5-1.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/1/5-2.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Our model\n",
    "### 1. Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_create_inputs():\n",
    "    global X_train, y_train, h_state_input, dropout, batch_size\n",
    "\n",
    "    X_train = tf.placeholder(tf.uint8, [None, None], name=\"X_train\")\n",
    "    y_train = tf.placeholder(tf.uint8, [None, None], name=\"y_train\")\n",
    "    h_state_input = tf.placeholder(tf.float32, [None, hStateSize * nLayers], name=\"h_state_input\")\n",
    "\n",
    "\n",
    "    dropout = tf.placeholder(tf.float32, name=\"dropout\")\n",
    "    batch_size = tf.placeholder(tf.int32, name=\"batch_size\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](./images/2/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "X_train = tf.placeholder(...)\n",
    "y_train = tf.placeholder(...)\n",
    "h_state_input = tf.placeholder(...)\n",
    "```\n",
    "![](./images/2/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Our model\n",
    "### 2. Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_create_architecture():\n",
    "    global rnn_cells_stack_outdropout\n",
    "\n",
    "    rnn_cells = [rnn.GRUCell(hStateSize) for _ in range(nLayers)]\n",
    "\n",
    "    rnn_cells_indropout = [\n",
    "        rnn.DropoutWrapper(cell, input_keep_prob=(1 - dropout))\n",
    "        for cell in rnn_cells\n",
    "    ]\n",
    "\n",
    "    rnn_cells_stack = rnn.MultiRNNCell(rnn_cells_indropout, state_is_tuple=False)\n",
    "\n",
    "    rnn_cells_stack_outdropout = rnn.DropoutWrapper(\n",
    "        rnn_cells_stack, output_keep_prob=(1 - dropout)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "rnn_cells = [rnn.GRUCell(hStateSize) for _ in range(nLayers)]\n",
    "```\n",
    "![](./images/3/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "rnn_cells_indropout = [rnn.DropoutWrapper(cell, input_keep_prob=(1 - dropout)) ...]\n",
    "```\n",
    "![](./images/3/2.png)\n",
    "\n",
    "_(Note: dropout only on inputs)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "rnn_cells_stack = rnn.MultiRNNCell(rnn_cells_indropout...)\n",
    "```\n",
    "![](./images/3/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    " rnn_cells_stack_outdropout = rnn.DropoutWrapper(\n",
    "    ..., output_keep_prob=(1 - dropout)\n",
    " )\n",
    "```\n",
    "![](./images/3/4.png)\n",
    "\n",
    "_(Note: dropout on output)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Our model\n",
    "### 3. TF Graph  (1/3) - RNN Forward Propagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_graph_forward_propagation():\n",
    "    global h_state_outputs, next_h_state_input, X_train_oh, y_train_oh\n",
    "\n",
    "    X_train_oh = tf.one_hot(X_train, alphaSize, 1.0, 0.0)\n",
    "    y_train_oh = tf.one_hot(y_train, alphaSize, 1.0, 0.0)\n",
    "\n",
    "    h_state_outputs, next_h_state_input = tf.nn.dynamic_rnn(\n",
    "        rnn_cells_stack_outdropout,\n",
    "        X_train_oh,\n",
    "        dtype=tf.float32,\n",
    "        initial_state=h_state_input\n",
    "    )\n",
    " \n",
    "    # named just to be able to use it later, when we restore the graph from disk\n",
    "    # to generate sequences\n",
    "    next_h_state_input = tf.identity(next_h_state_input, name='next_h_state_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "X_train_oh = tf.one_hot(X_train, alphaSize, 1.0, 0.0)\n",
    "y_train_oh = tf.one_hot(y_train, alphaSize, 1.0, 0.0)\n",
    "```\n",
    "![](./images/4/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "h_state_outputs, next_h_state_input = tf.nn.dynamic_rnn(\n",
    "    rnn_cells_stack_outdropout,\n",
    "    X_train_oh,\n",
    "    initial_state=h_state_input\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/4/2.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Our model\n",
    "### 3. TF Graph  (2/3) - Classification (Fully Connected Net + Softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_graph_fullyconnected_softmax():\n",
    "    global h_state_outputs_flat, y_preds_logit, y_preds_prob\n",
    "    \n",
    "    # flatting h_state_outputs\n",
    "    h_state_outputs_flat = tf.reshape(h_state_outputs, [-1, hStateSize])\n",
    "\n",
    "    y_preds_logit = layers.fully_connected(\n",
    "        h_state_outputs_flat,\n",
    "        alphaSize,\n",
    "        activation_fn=tf.nn.relu # the default\n",
    "        # activation_fn=tf.nn.softmax => WARNING: This op expects unscaled logits,\n",
    "        # since it performs a softmax on logits internally for efficiency.\n",
    "        # https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/softmax_cross_entropy_with_logits_v2\n",
    "    )\n",
    "\n",
    "    y_preds_prob = tf.nn.softmax(y_preds_logit, name=\"y_preds_prob\")  # (BS*MSL, AS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Classification\n",
    "```python\n",
    "y_preds_prob = tf.nn.softmax(y_preds_logit)\n",
    "```\n",
    "![](./images/5/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "h_state_outputs_flat = tf.reshape(h_state_outputs, [-1, hStateSize])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/5/2.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "y_preds_logit = layers.fully_connected(h_state_outputs_flat,alphaSize)\n",
    "```\n",
    "![](./images/5/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Our model\n",
    "### 3. TF Graph  (3/3) -  Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_graph_training():\n",
    "    global y_train_oh_flat, loss, train_step\n",
    "\n",
    "    y_train_oh_flat = tf.reshape(y_train_oh, [-1, alphaSize])\n",
    "\n",
    "    loss = tf.nn.softmax_cross_entropy_with_logits(logits=y_preds_logit, labels=y_train_oh_flat)\n",
    "    loss = tf.reshape(loss, [batch_size, -1])\n",
    "    \n",
    "    # Backpropagation\n",
    "    # Adam paper: https://arxiv.org/pdf/1412.6980.pdf\n",
    "    train_step = tf.train.AdamOptimizer(learningRate).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "y_train_oh_flat = tf.reshape(y_train_oh, [-1, alphaSize])\n",
    "loss = tf.nn.softmax_cross_entropy_with_logits(\n",
    "    logits=y_preds_logit, labels=y_train_oh_flat\n",
    ")\n",
    "```\n",
    "![](./images/6/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "loss = tf.reshape(loss, [batch_size, -1])\n",
    "train_step = tf.train.AdamOptimizer(learningRate).minimize(loss)\n",
    "```\n",
    "![](./images/6/2.png)\n",
    "\n",
    "_(backward propagation of errors)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. Training (1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def tf_init_session():\n",
    "    config = tf.ConfigProto()\n",
    "    config.gpu_options.allow_growth = True # only needed when using GPU\n",
    "    sess = tf.Session(config=config)\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    return sess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. Training (2/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def train(pattern_path, trained_model_name, n_epochs=20):\n",
    "    from time import time\n",
    "\n",
    "    dataset = Dataset.load_from_files(pattern_path)\n",
    "    sess = tf_init_session()\n",
    "    n_chars_processed = 0\n",
    "    oepoch = -1\n",
    "    t_start = time()\n",
    "\n",
    "    # initial input hidden state (zero)\n",
    "    in_h_state = np.zeros([batchSize, hStateSize * nLayers])\n",
    "    try:        \n",
    "        print(\"[training] starting to learn :) ...\")\n",
    "        for epoch, X_train_batch, y_train_batch in Dataset.get_training_batches(\n",
    "            n_epochs, batchSize, maxSeqLength, dataset\n",
    "        ):\n",
    "\n",
    "            _, next_in_h_state = sess.run(\n",
    "                [train_step, next_h_state_input],\n",
    "                feed_dict={\n",
    "                    X_train: X_train_batch, y_train: y_train_batch, h_state_input: in_h_state,\n",
    "                    dropout: dropoutProb, batch_size: batchSize\n",
    "                }\n",
    "            )\n",
    "            in_h_state = next_in_h_state\n",
    "\n",
    "            n_chars_processed += batchSize * maxSeqLength\n",
    "            if oepoch != epoch:\n",
    "                oepoch = epoch\n",
    "                print(\n",
    "                    \"[training][epoch %d/%d] characters processed: %d\" %\n",
    "                    (epoch + 1, n_epochs, n_chars_processed)\n",
    "                )\n",
    "    except KeyboardInterrupt:\n",
    "        pass\n",
    "\n",
    "    print(\"[training] training finished (%.1f mins)\" % ((time() - t_start) / 60))\n",
    "    saved_file = tf.train.Saver().save(sess, 'trained_models/test/%s' % trained_model_name)\n",
    "    print(\"[training] model saved: \" + saved_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 4. Training (2/2) - The training loop\n",
    "```python\n",
    "# initial input hidden state (zero)\n",
    "in_h_state = np.zeros([batchSize, hStateSize * nLayers])\n",
    "\n",
    "for epoch, X_train_batch, y_train_batch in Dataset.get_training_batches(...):\n",
    "\n",
    "    _, next_in_h_state = sess.run(\n",
    "        [train_step, next_h_state_input],\n",
    "        feed_dict={\n",
    "            X_train: X_train_batch, y_train: y_train_batch, h_state_input: in_h_state,\n",
    "            dropout: dropoutProb, batch_size: batchSize\n",
    "        }\n",
    "    )\n",
    "\n",
    "    in_h_state = next_in_h_state\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "for epoch, X_train_batch, y_train_batch in Dataset.get_training_batches(\n",
    "            n_epochs, batchSize, maxSeqLength, dataset\n",
    "    ):\n",
    "```\n",
    "![](./images/7/1.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. Sequence Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def generate_seq(model_path, seq_start=\"a\", length=1000):\n",
    "    tf.reset_default_graph()\n",
    "\n",
    "    output_seq = seq_start    \n",
    "    sess = tf_init_session()\n",
    "    with sess:\n",
    "        try:\n",
    "            saver = tf.train.import_meta_graph(model_path + '.meta')\n",
    "        except OSError:\n",
    "            print(\"[generate_seq] model %s does not exist\" % model_path)\n",
    "            return seq_start\n",
    "\n",
    "        saver.restore(sess, model_path)\n",
    "\n",
    "        in_X = Dataset.encode_str(output_seq)\n",
    "        in_X = np.array([in_X])  # \"1 bach of 1 sequence\"\n",
    "        \n",
    "        # initial input hidden state (zero)\n",
    "        in_h_state = np.zeros([1, hStateSize * nLayers], dtype=np.float32)\n",
    "        for i in range(length):\n",
    "            y_preds_prob, next_in_h_state = sess.run(\n",
    "                ['y_preds_prob:0', 'next_h_state_input:0'],\n",
    "                feed_dict={\n",
    "                    'X_train:0': in_X,\n",
    "                    'h_state_input:0': in_h_state,\n",
    "                    'dropout:0': 0., 'batch_size:0': 1\n",
    "                }\n",
    "            )\n",
    "            in_h_state = next_in_h_state\n",
    "\n",
    "            char = Dataset.peek_char_from_prob(y_preds_prob[-1], top_n=2)\n",
    "            in_X = np.array([[char]])  # \"1 bacth of 1 sequence of 1 char\"\n",
    "\n",
    "            output_seq += Dataset.decode_char(char)\n",
    "        return output_seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Generator loop\n",
    "```python\n",
    "# initial input hidden state (zero)\n",
    "in_h_state = np.zeros([1, hStateSize * nLayers], dtype=np.float32)\n",
    "for i in range(length):\n",
    "    y_preds_prob, next_in_h_state = sess.run(\n",
    "        ['y_preds_prob:0', 'next_h_state_input:0'],\n",
    "        feed_dict={\n",
    "            'X_train:0': in_X,\n",
    "            'h_state_input:0': in_h_state,\n",
    "            'dropout:0': 0., 'batch_size:0': 1\n",
    "        }\n",
    "    )\n",
    "    in_h_state = next_in_h_state\n",
    "    char = Dataset.peek_char_from_prob(y_preds_prob[-1], top_n=2)\n",
    "    in_X = np.array([[char]])  # \"1 bacth of 1 sequence of 1 char\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 5. Gluing it all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# This function will give life to our RNN\n",
    "def tf_work_your_magic():\n",
    "    tf.reset_default_graph() # we're gonna make magic more than once! XD\n",
    "\n",
    "    tf_create_inputs()\n",
    "    tf_create_architecture()\n",
    "    tf_graph_forward_propagation()\n",
    "    tf_graph_fullyconnected_softmax()\n",
    "    tf_graph_training()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let the fun begin..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Learning to \"payar\" (going The Martin Fierro's way).\n",
    "* Learning to compose some music.\n",
    "![](./images/pereyra_mate.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to _payar_!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./images/pereyra_guitar.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Aquí me pongo a cantar<br>\n",
    "al compás de la vigüela,<br>\n",
    "que el hombre que lo desvela<br>\n",
    "una pena estrordinaria,<br>\n",
    "como la ave solitaria<br>\n",
    "con el cantar se consuela.<br>\n",
    "<br>\n",
    "Pido a los santos del cielo<br>\n",
    "que ayuden mi pensamiento:<br>\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to _payar_!\n",
    "### 1.1. First things  first - preprocessing\n",
    "\n",
    "[dataset/martinfierro](dataset/martinfierro/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "mfierro_path = \"dataset/martinfierro/\"\n",
    "mfierro_files = mfierro_path + \"*.txt\"\n",
    "\n",
    "# NOTE: We need to run this code below only once\n",
    "Dataset.encode_files(mfierro_files, target_codec=\"utf8\")\n",
    "Dataset.normalize_files(mfierro_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[dataset/martinfierro](dataset/martinfierro/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to _payar_!\n",
    "### 1.2. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# making sure our RNN is alive\n",
    "tf_work_your_magic()\n",
    "\n",
    "\n",
    "train(mfierro_files, input(\"model name: \"), n_epochs=5)\n",
    "\n",
    "# NOTE: if it's taking too long, just press:\n",
    "#       <Esc> and then <I> twice to interrupt\n",
    "#       the process and save the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to _payar_!\n",
    "### 1.3. Payando :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "canto = generate_seq(\n",
    "    model_path=input(\"Trained model path: \"),\n",
    "    seq_start=input(\"Sequence starts with: \"),\n",
    "    length=1000\n",
    ")\n",
    "print(canto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to _payar_!\n",
    "### 1.4. Pretrained Payadores :D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from re import sub as re_replace\n",
    "\n",
    "trained_epochs = [0, 50, 400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "trained_model = \"trained_models/martinfierro/%depochs\" % trained_epochs[-1]\n",
    "\n",
    "canto = generate_seq(model_path=trained_model, seq_start=input(\"Starts with: \"), length=1000)\n",
    "\n",
    "canto = re_replace(r'\\d:\\s', '', canto) # deleting all :1,:2,:3... etc.\n",
    "print(canto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Let's learn how to compose music!\n",
    "### ( Now we're talking :D )\n",
    "![](./images/abc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The ABC Notation (http://abcnotation.com/)\n",
    "\n",
    "Example:\n",
    "```\n",
    "X: 1\n",
    "T: Cooley's\n",
    "M: 4/4\n",
    "L: 1/8\n",
    "R: reel\n",
    "K: Emin\n",
    "Q:120\n",
    "|:D2|EB{c}BA B2 EB|~B2 AB dBAG|FDAD BDAD|FDAD dAFD|\n",
    "EBBA B2 EB|B2 AB defg|afe^c dBAF|DEFD E2:|\n",
    "|:gf|eB B2 efge|eB B2 gedB|A2 FA DAFA|A2 FA defg|\n",
    "eB B2 eBgB|eB B2 defg|afe^c dBAF|DEFD E2:|\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Good tutorial:** [How to interpret abc music notation](http://www.lesession.co.uk/abc/abc_notation.htm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Playing ABC files\n",
    "\n",
    "\n",
    "**Software:** http://abcnotation.com/software\n",
    "\n",
    "**Webplayer:** https://abcjs.net/abcjs-editor.html\n",
    "\n",
    "\n",
    "**But** we're  gonna be using these commands: [abcmidi](http://abc.sourceforge.net/abcMIDI/original/), [timidity](https://sfxpt.wordpress.com/2015/02/02/how-to-play-midi-files-under-ubuntu-linux/).\n",
    "\n",
    "**Installation:**\n",
    "```\n",
    "~$ sudo apt install abcmidi\n",
    "~$ sudo apt-get install timidity timidity-interfaces-extra\n",
    "```\n",
    "**Usage:**\n",
    "```\n",
    "~$ abc2midi song.abc -o song.mid\n",
    "~$ timidity song.mid\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Let's learn how to compose music!\n",
    "### 2.1. Remember, first things  first - preprocessing\n",
    "\n",
    "[dataset/music](dataset/music/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "music_path = \"dataset/music/\"\n",
    "music_files = music_path + \"**/*.[ta][xb][tc]\"  # Recursively, all .abc or .txt files\n",
    "\n",
    "# NOTE: We need to run this code below only once\n",
    "Dataset.encode_files(music_files, target_codec=\"utf8\")\n",
    "Dataset.normalize_files(music_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Let's learn how to compose music!\n",
    "### 2.2. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# making sure our RNN is alive\n",
    "tf_work_your_magic()\n",
    "\n",
    "train(music_files, input(\"model name: \"), n_epochs=1)\n",
    "\n",
    "# WARNING: this dataset is 12,703,923 chars long...\n",
    "#          so each epoch is going to take a while.\n",
    "\n",
    "# NOTE: if it's taking too long, just press:\n",
    "#       <Esc> and then <I> twice to interrupt\n",
    "#       the process and save the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Let's dance!\n",
    "### 2.3. Dancing like a robot (a tiny little baby robot) :D\n",
    "\n",
    "But first, we need a DJ:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def play(str_abc):\n",
    "    from os import system\n",
    "    tmp_song = \"_tmp_song.abc\"\n",
    "    with open(tmp_song, \"w\") as abc_song:\n",
    "        abc_song.write(str_abc)\n",
    "\n",
    "    system('bash play_abc.bash ' + tmp_song)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Let's dance!\n",
    "### 2.3. Dancing like a robot (a tiny little baby robot) :D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "song = generate_seq(\n",
    "    model_path=input(\"Trained model path: \"),\n",
    "    seq_start=input(\"Sequence starts with: \"),\n",
    "    length=1000\n",
    ")\n",
    "print(song)\n",
    "\n",
    "play(song)\n",
    "# NOTE: press <Esc> and then <I> twice to stop playing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Let's dance!\n",
    "### 2.3. Dancing like a Pretrained Robot XD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from re import sub as re_replace\n",
    "\n",
    "trained_epochs = [0, 5, 10, 15, 20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "trained_model = \"trained_models/music/%depochs\" % trained_epochs[-1]\n",
    "\n",
    "song = generate_seq(model_path=trained_model, seq_start=\"X:\", length=1500)\n",
    "\n",
    "print(song)\n",
    "\n",
    "play(song) # (shake it)^n\n",
    "\n",
    "# NOTE: press <Esc> and then <I> twice to stop playing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thanks for you attention!\n",
    "### (...and that's it)\n",
    "![](./images/robot-dancing.gif)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python_3-TensorFlow",
   "language": "python",
   "name": "python_3-tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}