7.4one_layer_neural_network_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0110ENSkillsNetwork952-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Neural Networks with One Hidden Layer</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Objective</h2><ul><li> How to classify handwritten digits using Neural Network.</li></ul> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<p>In this lab, you will use a single layer neural network to classify handwritten digits from the MNIST database.</p>\n",
    "\n",
    "<ul>\n",
    "    <li><a href=\"#Model\">Neural Network Module and Training Function</a></li>\n",
    "    <li><a href=\"#Makeup_Data\">Make Some Data</a></li>\n",
    "    <li><a href=\"#Train\">Define the Neural Network, Optimizer, and Train the  Model</a></li>\n",
    "    <li><a href=\"#Result\">Analyze Results</a></li>\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong></p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll need the following libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import the libraries we need for this lab\n",
    "\n",
    "# Using the following line code to install the torchvision library\n",
    "# !mamba install -y torchvision\n",
    "\n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as dsets\n",
    "import torch.nn.functional as F\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the following helper functions for plotting the loss: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a function to plot accuracy and loss\n",
    "\n",
    "def plot_accuracy_loss(training_results): \n",
    "    plt.subplot(2, 1, 1)\n",
    "    plt.plot(training_results['training_loss'], 'r')\n",
    "    plt.ylabel('loss')\n",
    "    plt.title('training loss iterations')\n",
    "    plt.subplot(2, 1, 2)\n",
    "    plt.plot(training_results['validation_accuracy'])\n",
    "    plt.ylabel('accuracy')\n",
    "    plt.xlabel('epochs')   \n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the following function for printing the model parameters: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a function to plot model parameters\n",
    "\n",
    "def print_model_parameters(model):\n",
    "    count = 0\n",
    "    for ele in model.state_dict():\n",
    "        count += 1\n",
    "        if count % 2 != 0:\n",
    "            print (\"The following are the parameters for the layer \", count // 2 + 1)\n",
    "        if ele.find(\"bias\") != -1:\n",
    "            print(\"The size of bias: \", model.state_dict()[ele].size())\n",
    "        else:\n",
    "            print(\"The size of weights: \", model.state_dict()[ele].size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the neural network module or class: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a function to display data\n",
    "\n",
    "def show_data(data_sample):\n",
    "    plt.imshow(data_sample.numpy().reshape(28, 28), cmap='gray')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Model\">Neural Network Module and Training Function</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the neural network module or class: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a Neural Network class\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    # Constructor\n",
    "    def __init__(self, D_in, H, D_out):\n",
    "        super(Net, self).__init__()\n",
    "        self.linear1 = nn.Linear(D_in, H)\n",
    "        self.linear2 = nn.Linear(H, D_out)\n",
    "\n",
    "    # Prediction    \n",
    "    def forward(self, x):\n",
    "        x = torch.sigmoid(self.linear1(x))  \n",
    "        x = self.linear2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to train the model. In this case, the function returns a Python dictionary to store the training loss and accuracy on the validation data. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a training function to train the model\n",
    "\n",
    "def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100):\n",
    "    i = 0\n",
    "    useful_stuff = {'training_loss': [],'validation_accuracy': []}  \n",
    "    for epoch in range(epochs):\n",
    "        for i, (x, y) in enumerate(train_loader): \n",
    "            optimizer.zero_grad()\n",
    "            z = model(x.view(-1, 28 * 28))\n",
    "            loss = criterion(z, y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "             #loss for every iteration\n",
    "            useful_stuff['training_loss'].append(loss.data.item())\n",
    "        correct = 0\n",
    "        for x, y in validation_loader:\n",
    "            #validation \n",
    "            z = model(x.view(-1, 28 * 28))\n",
    "            _, label = torch.max(z, 1)\n",
    "            correct += (label == y).sum().item()\n",
    "        accuracy = 100 * (correct / len(validation_dataset))\n",
    "        useful_stuff['validation_accuracy'].append(accuracy)\n",
    "    return useful_stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Make Some Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train</code> to <code>True</code> and convert it to a tensor by placing a transform object in the argument <code>transform</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Processing...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/pip-req-build-w9kte7xz/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n"
     ]
    }
   ],
   "source": [
    "# Create training dataset\n",
    "\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset and convert it to a tensor by placing a transform object in the argument <code>transform</code>:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create validating dataset\n",
    "\n",
    "validation_dataset = dsets.MNIST(root='./data', download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the criterion function:  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create criterion function\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the training-data loader and the validation-data loader objects: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create data loader for both train dataset and valdiate dataset\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True)\n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Train\">Define the Neural Network, Optimizer, and Train the Model</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the model with 100 neurons: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model with 100 neurons\n",
    "\n",
    "input_dim = 28 * 28\n",
    "hidden_dim = 100\n",
    "output_dim = 10\n",
    "\n",
    "model = Net(input_dim, hidden_dim, output_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the model parameters: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the parameters for model\n",
    "\n",
    "print_model_parameters(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the optimizer object with a learning rate of 0.01: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate and the optimizer\n",
    "\n",
    "learning_rate = 0.01\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model by using 100 epochs **(this process takes time)**: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "\n",
    "training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Result\">Analyze Results</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training total loss or cost for every iteration and plot the training accuracy for every epoch:  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the accuracy and loss\n",
    "\n",
    "plot_accuracy_loss(training_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the first five misclassified samples:   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the first five misclassified samples\n",
    "\n",
    "count = 0\n",
    "for x, y in validation_dataset:\n",
    "    z = model(x.reshape(-1, 28 * 28))\n",
    "    _,yhat = torch.max(z, 1)\n",
    "    if yhat != y:\n",
    "        show_data(x)\n",
    "        count += 1\n",
    "    if count >= 5:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Practice</h3> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use <code>nn.Sequential</code> to build exactly the same model as you just built. Use the function <train>train</train> to train the model and use the function <code>plot_accuracy_loss</code> to see the metrics. Also, try different epoch numbers. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Practice: Use nn.Sequential to build the same model. Use plot_accuracy_loss to print out the accuarcy and loss\n",
    "\n",
    "# Type your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click <b>here</b> for the solution.\n",
    "\n",
    "<!-- \n",
    "input_dim = 28 * 28\n",
    "hidden_dim = 100\n",
    "output_dim = 10\n",
    "\n",
    "model = torch.nn.Sequential(\n",
    "    torch.nn.Linear(input_dim, hidden_dim),\n",
    "    torch.nn.Sigmoid(),\n",
    "    torch.nn.Linear(hidden_dim, output_dim),\n",
    ")\n",
    "learning_rate = 0.01\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n",
    "training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs = 10)\n",
    "plot_accuracy_loss(training_results)\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0110ENSkillsNetwork952-2022-01-01&context=cpdaas&apps=data_science_experience%2Cwatson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0110ENSkillsNetwork952-2022-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDL0110ENSkillsNetwork952-2022-01-01\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Change Log\n",
    "\n",
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2020-09-23  | 2.0  | Shubham  |  Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}