conv-lsuv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JguZ3NBZUkpo"
   },
   "source": [
    "# Training Convolutional Neural Networks (convnets or CNNs) with tf.keras and LSUV initilization.\n",
    "\n",
    "LSUV initialization proposed by Dmytro Mishkin and Jiri Matas in the article [All you need is a good Init](https://arxiv.org/pdf/1511.06422.pdf) consists of the two steps. \n",
    " - First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. \n",
    " - Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one.\n",
    "\n",
    "Original implementation can be found at [ducha-aiki/LSUVinit](https://github.com/ducha-aiki/LSUVinit)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import time\n",
    "from lib import LSUVinitialize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_SIZE = len(train_images)\n",
    "TEST_SIZE = len(test_images)\n",
    "\n",
    "train_images = np.asarray(train_images, dtype=np.float32) / 255\n",
    "\n",
    "# Convert the train images and add channels\n",
    "train_images = train_images.reshape((TRAINING_SIZE, 28, 28, 1))\n",
    "\n",
    "test_images = np.asarray(test_images, dtype=np.float32) / 255\n",
    "# Convert the train images and add channels\n",
    "test_images = test_images.reshape((TEST_SIZE, 28, 28, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How many categories we are predicting from (0-9)\n",
    "LABEL_DIMENSIONS = 10\n",
    "\n",
    "train_labels  = tf.keras.utils.to_categorical(train_labels, LABEL_DIMENSIONS)\n",
    "test_labels = tf.keras.utils.to_categorical(test_labels, LABEL_DIMENSIONS)\n",
    "\n",
    "# Cast the labels to floats, needed later\n",
    "train_labels = train_labels.astype(np.float32)\n",
    "test_labels = test_labels.astype(np.float32)\n",
    "\n",
    "class_names = [\n",
    "    'T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt',\n",
    "    'Sneaker', 'Bag', 'Ankle boot'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 11, 11, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 3, 3, 64)          36928     \n",
      "=================================================================\n",
      "Total params: 55,744\n",
      "Trainable params: 55,744\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "# model.add(tf.keras.Input(shape=train_images.shape[1:]))\n",
    "model.add(tf.keras.layers.Conv2D(filters=32, kernel_size=(3, 3), activation=tf.nn.relu, input_shape=(28, 28, 1)))\n",
    "model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=2))\n",
    "model.add(tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=2))\n",
    "model.add(tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), activation=tf.nn.relu))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see above that the output of every convolutional layer is a 3D tensor of shape (`height`, `width`, `filters`). The width and height tend to get smaller as we go deeper into the network and the number of filters or channels increases from the input channel size of 1. \n",
    "\n",
    "The last part of the network for the classification task is similar to the other notebooks and consists of `Dense` layers which process 1D vectors. So we first need to `Flatten` our 3D outputs from the convolutional part to 1D and then add the `Dense` layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(tf.keras.layers.Dense(64, activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.Dense(LABEL_DIMENSIONS, activation=tf.nn.softmax))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the network is again similar to all the previous notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 11, 11, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 3, 3, 64)          36928     \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 576)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 64)                36928     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 93,322\n",
      "Trainable params: 93,322\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=optimizer,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE=128\n",
    "\n",
    "# Because tf.data may work with potentially **large** collections of data\n",
    "# we do not shuffle the entire dataset by default\n",
    "# Instead, we maintain a buffer of SHUFFLE_SIZE elements\n",
    "# and sample from there.\n",
    "SHUFFLE_SIZE = 10000 \n",
    "\n",
    "# Create the dataset\n",
    "training_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))\n",
    "training_dataset = training_dataset.shuffle(SHUFFLE_SIZE)\n",
    "training_dataset = training_dataset.batch(BATCH_SIZE).repeat()\n",
    "\n",
    "validation_dataset = tf.data.Dataset.from_tensor_slices((test_images, test_labels))\n",
    "validation_dataset = validation_dataset.shuffle(SHUFFLE_SIZE)\n",
    "validation_dataset = validation_dataset.batch(BATCH_SIZE).repeat()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def svd_orthonormal(shape):\n",
    "    # Orthonorm init code is taked from Lasagne\n",
    "    # https://github.com/Lasagne/Lasagne/blob/master/lasagne/init.py\n",
    "    if len(shape) < 2:\n",
    "        raise RuntimeError(\"Only shapes of length 2 or more are supported.\")\n",
    "    flat_shape = (shape[0], np.prod(shape[1:]))\n",
    "    a = np.random.standard_normal(flat_shape)\n",
    "    u, _, v = np.linalg.svd(a, full_matrices=False)\n",
    "    q = u if u.shape == flat_shape else v\n",
    "    q = q.reshape(shape)\n",
    "    return q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Init Layer conv2d\n",
      "0.06591233\n",
      "Init Layer conv2d_1\n",
      "0.18357301\n",
      "Init Layer conv2d_2\n",
      "0.2051178\n",
      "Init Layer dense\n",
      "0.62251633\n",
      "dense_1 too small\n",
      "LSUV: total layers initialized 4\n"
     ]
    }
   ],
   "source": [
    "model = LSUVinitialize(model, train_images[:BATCH_SIZE,:,:,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 31 steps, validate for 31 steps\n",
      "Epoch 1/14\n",
      "31/31 [==============================] - 4s 127ms/step - loss: 1.4361 - accuracy: 0.5479 - val_loss: 0.8141 - val_accuracy: 0.7026\n",
      "Epoch 2/14\n",
      "31/31 [==============================] - 3s 85ms/step - loss: 0.7314 - accuracy: 0.7273 - val_loss: 0.6603 - val_accuracy: 0.7576\n",
      "Epoch 3/14\n",
      "31/31 [==============================] - 3s 84ms/step - loss: 0.6382 - accuracy: 0.7689 - val_loss: 0.5787 - val_accuracy: 0.7923\n",
      "Epoch 4/14\n",
      "31/31 [==============================] - 3s 81ms/step - loss: 0.5715 - accuracy: 0.7918 - val_loss: 0.5693 - val_accuracy: 0.7886\n",
      "Epoch 5/14\n",
      "31/31 [==============================] - 2s 81ms/step - loss: 0.5260 - accuracy: 0.8082 - val_loss: 0.5614 - val_accuracy: 0.7954\n",
      "Epoch 6/14\n",
      "31/31 [==============================] - 3s 84ms/step - loss: 0.5113 - accuracy: 0.8087 - val_loss: 0.4834 - val_accuracy: 0.8296\n",
      "Epoch 7/14\n",
      "31/31 [==============================] - 2s 75ms/step - loss: 0.4559 - accuracy: 0.8387 - val_loss: 0.4696 - val_accuracy: 0.8254\n",
      "Epoch 8/14\n",
      "31/31 [==============================] - 2s 78ms/step - loss: 0.4549 - accuracy: 0.8392 - val_loss: 0.4679 - val_accuracy: 0.8347\n",
      "Epoch 9/14\n",
      "31/31 [==============================] - 2s 79ms/step - loss: 0.4431 - accuracy: 0.8460 - val_loss: 0.4400 - val_accuracy: 0.8387\n",
      "Epoch 10/14\n",
      "31/31 [==============================] - 2s 73ms/step - loss: 0.4306 - accuracy: 0.8425 - val_loss: 0.4467 - val_accuracy: 0.8402\n",
      "Epoch 11/14\n",
      "31/31 [==============================] - 2s 73ms/step - loss: 0.4056 - accuracy: 0.8606 - val_loss: 0.4751 - val_accuracy: 0.8264\n",
      "Epoch 12/14\n",
      "31/31 [==============================] - 2s 76ms/step - loss: 0.4190 - accuracy: 0.8493 - val_loss: 0.4332 - val_accuracy: 0.8339\n",
      "Epoch 13/14\n",
      "31/31 [==============================] - 2s 75ms/step - loss: 0.4010 - accuracy: 0.8488 - val_loss: 0.4232 - val_accuracy: 0.8425\n",
      "Epoch 14/14\n",
      "31/31 [==============================] - 2s 73ms/step - loss: 0.3745 - accuracy: 0.8695 - val_loss: 0.3908 - val_accuracy: 0.8548\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14667af90>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 14\n",
    "\n",
    "model.fit(training_dataset, \n",
    "    validation_data=validation_dataset,\n",
    "    steps_per_epoch = 1000 // 32,\n",
    "    epochs=EPOCHS,\n",
    "    validation_steps=1000 // 32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again to evaluate the model we need to check the accuracy on unseen or test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Model   Loss: 0.395358\tAccuracy: 0.856500\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=0)\n",
    "print('\\nTest Model   Loss: %.6f\\tAccuracy: %.6f' % (test_loss, test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "5-conv-draft.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}