Problem_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 2: Getting familiar with TensorFlow\n",
    "\n",
    "*TensorFlow* is one of the most popular deep learning framework developed by Google. If you are new to TensorFlow, please read and play with the sample in [Getting started with TensorFlow](https://www.tensorflow.org/get_started/get_started) to get started.\n",
    "\n",
    "* <b>Learning Objective:</b> In Problem 1, you implemented a fully connected network from scratch on your own. Very tedious to do it all by yourself, right? Well, we actually feel the same thing, that's why we are using tools instead of doing everything from scratch, lonely and depressed. For this part of the assignment, we will familiarize you with a widely-used deep learning framework developed by Google, TensorFlow and walk you through convolutional neural networks and show how to train them.\n",
    "* <b>Provided Codes:</b> We provide the Template class for a simple CNN model as BaseModel, predefined skeletons for conv2d() and max_pool(), as well as the dataset preprocessing parts.\n",
    "* <b>TODOs:</b> You are asked to implement the BaseModel following the detailed instrunctions and design your own model in YourModel to achieve a reasonably good performance for classification task on CIFAR-10 dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install and import libraries\n",
    "Install tensorflow and matplotlib.\n",
    "\n",
    "```\n",
    "pip install -U tensorflow matplotlib\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "# Add whatever you want\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "print \"TensorFlow Version {}\".format(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Test matplotlib\n",
    "x = np.linspace(-3, 3, 100)\n",
    "\n",
    "plt.plot(x, np.maximum(0, x), label='relu')\n",
    "plt.plot(x, 1/(1 + np.exp(-x)), label='sigmoid')\n",
    "plt.plot(x, (1 - np.exp(-2 * x))/(1 + np.exp(-2 * x)), label='tanh')\n",
    "\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "\n",
    "plt.title(\"Activation functions\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Test tensorflow\n",
    "print('TensorFlow version: ' + tf.__version__)\n",
    "a = tf.constant(2.0)\n",
    "b = tf.constant(3.0)\n",
    "c = a * b\n",
    "\n",
    "sess = tf.Session()\n",
    "result = sess.run([a, b, c])\n",
    "print('%f * %f = %f' % (result[0], result[1], result[2]))\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load datasets\n",
    "Download [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz) and load the dataset. In this assignment, we will use all 50,000 images for training and 10,000 images for test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Configuration\n",
    "num_training = 49000\n",
    "num_validation = 50000 - num_training\n",
    "num_test = 10000\n",
    "\n",
    "def unpickle(file):\n",
    "    import sys\n",
    "    if sys.version_info.major == 2:\n",
    "        import cPickle\n",
    "        with open(file, 'rb') as fo:\n",
    "            dict = cPickle.load(fo)\n",
    "        return dict['data'], dict['labels']\n",
    "    else:\n",
    "        import pickle\n",
    "        with open(file, 'rb') as fo:\n",
    "            dict = pickle.load(fo, encoding='bytes')\n",
    "        return dict[b'data'], dict[b'labels']\n",
    "\n",
    "def load_train_data():\n",
    "    pass\n",
    "\n",
    "    #############################################################################\n",
    "    # TODO: Load training data from cifar-10 dataset                            #\n",
    "    # Load five files from 'data_batch_1' to 'data_batch_5'                     #\n",
    "    # Reshape images and labels to the shape of [50000, 32, 32, 3]              # \n",
    "    # and [50000], respectively                                                 #\n",
    "    #############################################################################\n",
    "\n",
    "    #############################################################################\n",
    "    #                             END OF YOUR CODE                              #\n",
    "    #############################################################################\n",
    "\n",
    "def load_test_data():\n",
    "    pass\n",
    "\n",
    "    #############################################################################\n",
    "    # TODO: Load testing data from cifar-10 dataset                             #\n",
    "    # Load 'test_batch' file                                                    #\n",
    "    # Reshape images and labels to the shape of [10000, 32, 32, 3]              #\n",
    "    # and [10000], respectively                                                 #\n",
    "    #############################################################################\n",
    "    \n",
    "    #############################################################################\n",
    "    #                             END OF YOUR CODE                              #\n",
    "    #############################################################################\n",
    "\n",
    "# Load cifar-10 data\n",
    "X_train, Y_train, X_val, Y_val = load_train_data()\n",
    "X_test, Y_test = load_test_data()\n",
    "\n",
    "# Check the shape of the dataset\n",
    "assert X_train.shape == (num_training, 32, 32, 3)\n",
    "assert Y_train.shape == (num_training, )\n",
    "assert X_val.shape == (num_validation, 32, 32, 3)\n",
    "assert Y_val.shape == (num_validation, )\n",
    "assert X_test.shape == (num_test, 32, 32, 3)\n",
    "assert Y_test.shape == (10000, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2-1\n",
    "\n",
    "Using the code provided, implement a neural network architecture with an optimization routine according to the specification provided below.\n",
    "\n",
    "**Model:**\n",
    "- Input image with the size 32x32x3\n",
    "- 7x7 convolutional layer with 32 filters, stride of 1, and padding 'SAME'\n",
    "- ReLU activation layer\n",
    "- 3x3 max pooling layer with a stride of 2\n",
    "- 5x5 convolutional layer with 64 filters, stride of 1, and padding 'SAME'\n",
    "- ReLU activation layer\n",
    "- 3x3 max pooling layer with a stride of 2\n",
    "- Flatten layer (8x8x64 -> 4096)\n",
    "- Fully-connected layer with 384 output units (4096 -> 384)\n",
    "- ReLU activation layer\n",
    "- Fully-connected layer with 10 output units (384 -> 10)\n",
    "- Output logits (10)\n",
    "\n",
    "**Optimizer:**\n",
    "- Adam optimizer\n",
    "\n",
    "**Learning rate:**\n",
    "- Set start learning rate as 5e-4 and apply exponential decay every 500 steps with a base of 0.96\n",
    "- Use 'tf.train.exponential_decay' and 'tf.train.AdamOptimizer'\n",
    "\n",
    "**Loss:**\n",
    "- Softmax cross entropy loss\n",
    "- Use 'tf.nn.softmax_cross_entropy_with_logits'\n",
    "\n",
    "\n",
    "Your model **should** achieve about 60% accuracy on validation set in 5 epochs using provided evaluation code.\n",
    "\n",
    "You can modify the template code as you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define your layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define max pooling and conv layers\n",
    "def conv2d(input, kernel_size, stride, num_filter):\n",
    "    stride_shape = [1, stride, stride, 1]\n",
    "    filter_shape = [kernel_size, kernel_size, input.get_shape()[3], num_filter]\n",
    "\n",
    "    W = tf.get_variable('w', filter_shape, tf.float32, tf.random_normal_initializer(0.0, 0.02))\n",
    "    b = tf.get_variable('b', [1, 1, 1, num_filter], initializer=tf.constant_initializer(0.0))\n",
    "    return tf.nn.conv2d(input, W, stride_shape, padding='SAME') + b\n",
    "\n",
    "def max_pool(input, kernel_size, stride):\n",
    "    ksize = [1, kernel_size, kernel_size, 1]\n",
    "    strides = [1, stride, stride, 1]\n",
    "    return tf.nn.max_pool(input, ksize=ksize, strides=strides, padding='SAME')\n",
    "\n",
    "#############################################################################\n",
    "# TODO: You can add any layers (fully-connected, normalization)             #\n",
    "#############################################################################\n",
    "\n",
    "#############################################################################\n",
    "#                             END OF YOUR CODE                              #\n",
    "#############################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample convolutional nueral network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class BaseModel(object):\n",
    "    def __init__(self):\n",
    "        self.num_epoch = 5\n",
    "        self.batch_size = 128\n",
    "        self.log_step = 50\n",
    "        self._build_model()\n",
    "\n",
    "    def _model(self):\n",
    "        print('-' * 5 + '  Sample model  ' + '-' * 5)\n",
    "\n",
    "        print('intput layer: ' + str(self.X.get_shape()))\n",
    "\n",
    "        with tf.variable_scope('conv1'):\n",
    "            self.conv1 = conv2d(self.X, 7, 1, 32)\n",
    "            self.relu1 = tf.nn.relu(self.conv1)\n",
    "            self.pool1 = max_pool(self.relu1, 3, 2)            \n",
    "            print('conv1 layer: ' + str(self.pool1.get_shape()))\n",
    "\n",
    "        with tf.variable_scope('conv2'):\n",
    "            #############################################################################\n",
    "            # TODO: Complete the following functions                                    #\n",
    "            #############################################################################\n",
    "            self.conv2 = None\n",
    "            self.relu2 = None\n",
    "            self.pool2 = None            \n",
    "            #############################################################################\n",
    "            #                             END OF YOUR CODE                              #\n",
    "            #############################################################################\n",
    "            print('conv2 layer: ' + str(self.pool2.get_shape()))\n",
    "\n",
    "\n",
    "        #############################################################################\n",
    "        # TODO: Flatten the output tensor from conv2 layer                          #\n",
    "        #############################################################################\n",
    "        self.flat = None\n",
    "        #############################################################################\n",
    "        #                             END OF YOUR CODE                              #\n",
    "        #############################################################################      \n",
    "        print('flat layer: ' + str(self.flat.get_shape()))\n",
    "\n",
    "        with tf.variable_scope('fc3'):\n",
    "            #############################################################################\n",
    "            # TODO: Complete the following functions                                    #\n",
    "            #############################################################################\n",
    "            self.fc3 = None\n",
    "            self.relu3 = None\n",
    "            #############################################################################\n",
    "            #                             END OF YOUR CODE                              #\n",
    "            #############################################################################\n",
    "            print('fc3 layer: ' + str(self.relu3.get_shape()))\n",
    "\n",
    "        with tf.variable_scope('fc4'):\n",
    "            #############################################################################\n",
    "            # TODO: Complete the following functions                                    #\n",
    "            #############################################################################\n",
    "            self.fc4 = None            \n",
    "            #############################################################################\n",
    "            #                             END OF YOUR CODE                              #\n",
    "            #############################################################################\n",
    "            print('fc4 layer: ' + str(self.fc4.get_shape()))\n",
    "        \n",
    "        # Return the last layer\n",
    "        return self.fc4\n",
    "\n",
    "    def _input_ops(self):\n",
    "        # Placeholders\n",
    "        self.X = tf.placeholder(tf.float32, [None, 32, 32, 3])\n",
    "        self.Y = tf.placeholder(tf.int64, [None])\n",
    "        \n",
    "        #############################################################################\n",
    "        # TODO: You can add any placeholders                                        #\n",
    "        #############################################################################\n",
    "        self.is_train = None\n",
    "        self.keep_prob = None\n",
    "        #############################################################################\n",
    "        #                             END OF YOUR CODE                              #\n",
    "        #############################################################################\n",
    "\n",
    "    def _build_optimizer(self):\n",
    "        # Adam optimizer 'self.train_op' that minimizes 'self.loss_op'\n",
    "        #############################################################################\n",
    "        # TODO: Complete the following functions                                    #\n",
    "        #############################################################################\n",
    "        self.train_op = None        \n",
    "        #############################################################################\n",
    "        #                             END OF YOUR CODE                              #\n",
    "        #############################################################################\n",
    "        \n",
    "    def _loss(self, labels, logits):\n",
    "        # Softmax cross entropy loss 'self.loss_op'\n",
    "        #############################################################################\n",
    "        # TODO: Complete the following functions                                    #\n",
    "        #############################################################################\n",
    "        self.loss_op = None        \n",
    "        #############################################################################\n",
    "        #                             END OF YOUR CODE                              #\n",
    "        #############################################################################\n",
    "\n",
    "    def _build_model(self):\n",
    "        # Define input variables\n",
    "        self._input_ops()\n",
    "\n",
    "        # Convert Y to one-hot vector\n",
    "        labels = tf.one_hot(self.Y, 10)\n",
    "\n",
    "        # Build a model and get logits\n",
    "        logits = self._model()\n",
    "\n",
    "        # Compute loss\n",
    "        self._loss(labels, logits)\n",
    "        \n",
    "        # Build optimizer\n",
    "        self._build_optimizer()\n",
    "\n",
    "        # Compute accuracy\n",
    "        predict = tf.argmax(logits, 1)\n",
    "        correct = tf.equal(predict, self.Y)\n",
    "        self.accuracy_op = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "        \n",
    "    def train(self, sess, X_train, Y_train, X_val, Y_val):\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "\n",
    "        step = 0\n",
    "        losses = []\n",
    "        accuracies = []\n",
    "        print('-' * 5 + '  Start training  ' + '-' * 5)\n",
    "        for epoch in range(self.num_epoch):\n",
    "            print('train for epoch %d' % epoch)\n",
    "            for i in range(num_training // self.batch_size):\n",
    "                X_ = X_train[i * self.batch_size:(i + 1) * self.batch_size][:]\n",
    "                Y_ = Y_train[i * self.batch_size:(i + 1) * self.batch_size]\n",
    "\n",
    "                #############################################################################\n",
    "                # TODO: You can change feed data as you want                                #\n",
    "                #############################################################################\n",
    "                feed_dict = None                \n",
    "                #############################################################################\n",
    "                #                             END OF YOUR CODE                              #\n",
    "                #############################################################################\n",
    "                fetches = [self.train_op, self.loss_op, self.accuracy_op]\n",
    "\n",
    "                _, loss, accuracy = sess.run(fetches, feed_dict=feed_dict)\n",
    "                losses.append(loss)\n",
    "                accuracies.append(accuracy)\n",
    "\n",
    "                if step % self.log_step == 0:\n",
    "                    print('iteration (%d): loss = %.3f, accuracy = %.3f' %\n",
    "                        (step, loss, accuracy))\n",
    "                step += 1\n",
    "\n",
    "            #############################################################################\n",
    "            # TODO: Plot training curves                                                #\n",
    "            #############################################################################\n",
    "            # Graph 1. X: epoch, Y: training loss\n",
    "\n",
    "            # Graph 2. X: epoch, Y: training accuracy\n",
    "            \n",
    "            #############################################################################\n",
    "            #                             END OF YOUR CODE                              #\n",
    "            #############################################################################\n",
    "\n",
    "            # Print validation results\n",
    "            print('validation for epoch %d' % epoch)\n",
    "            val_accuracy = self.evaluate(sess, X_val, Y_val)\n",
    "            print('-  epoch %d: validation accuracy = %.3f' % (epoch, val_accuracy))\n",
    "\n",
    "    def evaluate(self, sess, X_eval, Y_eval):\n",
    "        eval_accuracy = 0.0\n",
    "        eval_iter = 0\n",
    "        for i in range(X_eval.shape[0] // self.batch_size):\n",
    "            X_ = X_eval[i * self.batch_size:(i + 1) * self.batch_size][:]\n",
    "            Y_ = Y_eval[i * self.batch_size:(i + 1) * self.batch_size]\n",
    "                        \n",
    "            #############################################################################\n",
    "            # TODO: You can change feed data as you want                                #\n",
    "            #############################################################################\n",
    "            feed_dict = None\n",
    "            #############################################################################\n",
    "            #                             END OF YOUR CODE                              #\n",
    "            #############################################################################\n",
    "            accuracy = sess.run(self.accuracy_op, feed_dict=feed_dict)\n",
    "            eval_accuracy += accuracy\n",
    "            eval_iter += 1\n",
    "        return eval_accuracy / eval_iter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clear old computation graphs\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Train our sample model\n",
    "with tf.Session() as sess:\n",
    "    with tf.device('/cpu:0'):\n",
    "        model = BaseModel()\n",
    "        model.train(sess, X_train, Y_train, X_val, Y_val)\n",
    "        accuracy = model.evaluate(sess, X_test, Y_test)\n",
    "        print('***** test accuracy: %.3f' % accuracy)\n",
    "        saver = tf.train.Saver()\n",
    "        model_path = saver.save(sess, \"lib/tf_models/problem2/csci-599_sample.ckpt\")\n",
    "        print(\"Model saved in %s\" % model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2-2\n",
    "\n",
    "Implement your own model. \n",
    "\n",
    "You can modify the template code as you want and you can use GPU for fast training.\n",
    "\n",
    "These are the techniques that you can try:\n",
    "- Data preprocessing\n",
    "- Data augmentation\n",
    "- Dropout\n",
    "- Batch normalization\n",
    "- More convolutional layers\n",
    "- More training epochs\n",
    "- Learning rate decay\n",
    "- Any other models and techniqes\n",
    "\n",
    "Your model should achieve >= 70% accuracy on the test set of CIFAR-10.\n",
    "\n",
    "If the accuracy of the model reaches to 80% on the test set, you will get 5 extra points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class YourModel(BaseModel):\n",
    "    def __init__(self):\n",
    "        super(YourModel, self).__init__()\n",
    "        self.num_epoch = 10\n",
    "\n",
    "    def _model(self):\n",
    "        print('-' * 5 + '  Your model  ' + '-' * 5)\n",
    "\n",
    "        #############################################################################\n",
    "        # TODO: Implement you own model here                                        #\n",
    "        #############################################################################\n",
    "                \n",
    "        #############################################################################\n",
    "        #                             END OF YOUR CODE                              #\n",
    "        #############################################################################\n",
    "\n",
    "    #############################################################################\n",
    "    # TODO: You can redefine BaseModel's methods                                #\n",
    "    #############################################################################\n",
    "\n",
    "    #############################################################################\n",
    "    #                             END OF YOUR CODE                              #\n",
    "    #############################################################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clear old computation graphs\n",
    "tf.reset_default_graph()\n",
    "\n",
    "sess = tf.Session()\n",
    "\n",
    "#############################################################################\n",
    "# TODO: Preprocessing                                                       #\n",
    "#############################################################################\n",
    "X_train_ = X_train\n",
    "X_val_ = X_val\n",
    "X_test_ = X_test\n",
    "#############################################################################\n",
    "#                             END OF YOUR CODE                              #\n",
    "#############################################################################\n",
    "\n",
    "model = YourModel()\n",
    "model.train(sess, X_train_, Y_train, X_val_, Y_val)\n",
    "accuracy = model.evaluate(sess, X_test_, Y_test)\n",
    "print('***** test accuracy: %.3f' % accuracy)\n",
    "\n",
    "# Save your model\n",
    "saver = tf.train.Saver()\n",
    "model_path = saver.save(sess, \"lib/tf_models/problem2/csci-599_mine.ckpt\")\n",
    "print(\"Model saved in %s\" % model_path)\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "# Load your model\n",
    "model = YourModel()\n",
    "sess = tf.Session()\n",
    "saver = tf.train.Saver()\n",
    "saver.restore(sess, \"lib/tf_models/problem2/csci-599_mine.ckpt\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}