1B_MultiLayer_Perceptron_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Assignment: Multi-Layer Perceptron (MLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-Layer Perceptrons\n",
    "\n",
    "The simple logistic regression example we went over in the previous notebook is essentially a one-layer neural network, projecting straight from the input to the output predictions.\n",
    "While this can be effective for linearly separable data, occasionally a little more complexity is necessary.\n",
    "Neural networks with additional layers are typically able to learn more complex functions, leading to better performance.\n",
    "These additional layers (called \"hidden\" layers) transform the input into one or more intermediate representations before making a final prediction.\n",
    "\n",
    "In the logistic regression example, the way we performed the transformation was with a fully-connected layer, which consisted of a linear transform (matrix multiply plus a bias).\n",
    "A neural network consisting of multiple successive fully-connected layers is commonly called a Multi-Layer Perceptron (MLP). \n",
    "In the simple MLP below, a 4-d input is projected to a 5-d hidden representation, which is then projected to a single output that is used to make the final prediction.\n",
    "\n",
    "<img src=\"Figures/MLP.png\" width=\"300\"/>\n",
    "\n",
    "For the assignment, you will be building a MLP for MNIST.\n",
    "Mechanically, this is done very similary to our logistic regression example, but instead of going straight to a 10-d vector representing our output predictions, we might first transform to a 500-d vector with a \"hidden\" layer, then to the output of dimension 10.\n",
    "Before you do so, however, there's one more important thing to consider.\n",
    "\n",
    "### Nonlinearities\n",
    "\n",
    "We typically include nonlinearities between layers of a neural network.\n",
    "There's a number of reasons to do so.\n",
    "For one, without anything nonlinear between them, successive linear transforms (fully connected layers) collapse into a single linear transform, which means the model isn't any more expressive.\n",
    "On the other hand, intermediate nonlinearities prevent this collapse, allowing neural networks to approximate more complex functions.\n",
    "\n",
    "There are a number of nonlinearities commonly used in neural networks, but one of the most popular is the [rectified linear unit (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)):\n",
    "\n",
    "\\begin{align}\n",
    "x = \\max(0,x)\n",
    "\\end{align}\n",
    "\n",
    "There are a number of ways to implement this in PyTorch.\n",
    "We could do it with elementary PyTorch operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.rand(5, 3)*2 - 1\n",
    "x_relu_max = torch.max(torch.zeros_like(x),x)\n",
    "\n",
    "print(\"x: {}\".format(x))\n",
    "print(\"x after ReLU with max: {}\".format(x_relu_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, PyTorch also has the ReLU implemented, for example in `torch.nn.functional`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "x_relu_F = F.relu(x)\n",
    "\n",
    "print(\"x after ReLU with nn.functional: {}\".format(x_relu_F))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assignment\n",
    "\n",
    "Build a 2-layer MLP for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:\n",
    "\n",
    "Image (784 dimensions) ->  \n",
    "fully connected layer (500 hidden units) -> nonlinearity (ReLU) ->  \n",
    "fully connected (10 hidden units) -> softmax\n",
    "\n",
    "Try building the model both with basic PyTorch operations, and then again with more object-oriented higher-level APIs. \n",
    "You should get similar results!\n",
    "\n",
    "\n",
    "*Some hints*:\n",
    "- Even as we add additional layers, we still only require a single optimizer to learn the parameters.\n",
    "Just make sure to pass all parameters to it!\n",
    "- As you'll calculate in the Short Answer, this MLP model has many more parameters than the logisitic regression example, which makes it more challenging to learn.\n",
    "To get the best performance, you may want to play with the learning rate and increase the number of training epochs.\n",
    "- Be careful using `torch.nn.CrossEntropyLoss()`. \n",
    "If you look at the [PyTorch documentation](https://pytorch.org/docs/stable/nn.html#crossentropyloss): you'll see that `torch.nn.CrossEntropyLoss()` combines the softmax operation with the cross-entropy.\n",
    "This means you need to pass in the logits (predictions pre-softmax) to this loss.\n",
    "Computing the softmax separately and feeding the result into `torch.nn.CrossEntropyLoss()` will significantly degrade your model's performance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### YOUR CODE HERE\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Make sure to print out your accuracy on the test set at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short answer\n",
    "How many trainable parameters does your model have? \n",
    "How does this compare to the logisitic regression example?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}