1.4_lesson4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 4  of Raynard's Python Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are still with me at this point, I'm proud of you. And I'm proud that I can be at some help in this amazing moment of your coding career\n",
    "."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now I'm going to brief you with libs and ideas, after the take off, the rest of endeavor is yours and yours alone, I shouldn't deprive you of privilige to explore. So no more training wheels for pandas. (Also you can find out I'm starting to fail to answer many questions about pandas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today we will start to talk about numpy, the building blocks for machine learning.\n",
    "\n",
    "It deals with matrices and linear algebra.\n",
    "\n",
    "If you forget all the linear algebra, I strongly recommend the 3blue and 1brown video, a dozen of 10 min videos will give you intuitive understanding of linear algebra. University professors and nubies love it. And was promoted on many university courses\n",
    "\n",
    "[3 blue  and 1 brown click here](https://www.youtube.com/watch?v=kjBOesZCoqc&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)\n",
    "\n",
    "You can go on with code for now, and watch the video later\n",
    "\n",
    "Linear algebra is beutiful,  it represents transformation, and one hell of a way to do effective calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Its documents will tell you this \n",
    "a=np.array([[2,2,2],[1,1,1]])\n",
    "# a vector (1 dimensional matrix)\n",
    "x = np.matrix(\".3 .5 .6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# array is like the dataframe for numpy, every array instance will has attribute (like property) called shape\n",
    "a.shape,x.shape\n",
    "# shows the shape of a tensor (multi dimensional matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The Transpose of a \n",
    "a.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Matrix times a scaler (matrix scales according to a single value )\n",
    "a*5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "exercises:\n",
    "1. multiply all matrix elements by 10\n",
    "\n",
    "2. divide all matrix elements by 2\n",
    "\n",
    "3. multiply all matrix elements by 10 and change to integers\n",
    "there are 2 way to do that (change type and round)\n",
    "\n",
    "4. reverse all values of matrix (e.g. change 2 to -2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape is 2(rows) by 3(columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's generate some array with the shape specified but with all the random numbers\n",
    "b=np.random.rand(3,4)\n",
    "print(b)\n",
    "# please observe how numbers of dimensions are woven according to number 3,4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "c=np.matmul(a,b)\n",
    "# if the columns(or the last number) of a matrix(3 for a) equals the rows(or the first number) of a mitrix(3 for b).\n",
    "# You can multiply the 2 matrix\n",
    "# Check matrix multiplication for details\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luckily in python3 you can do this with your lazy ass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "c=a@b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Isn't that neat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d=np.random.rand(4,10)\n",
    "print(d.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A matrix can be reshaped to other matrix, as long as the total number of elements doesn't change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d.reshape(5,8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d.reshape(5,2,2,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or you can specify 1 dimension as -1, to let the numpy figure out the structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d.reshape(-1,5)\n",
    "# Numpy will find out the fisrt dimension is 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = np.array([[.32, .56, .63], [.26, .13, .97]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "exercise: \n",
    "\n",
    "Use matrix X and:\n",
    "\n",
    "1.  transpose matrix X\n",
    "\n",
    "2.  use check matrix shape (property .shape) \n",
    "\n",
    "3.  use function .reshape() and change matrix X shapes:\n",
    "\n",
    "    3a. shape [1, 6] (there are 3 ways you can do it)\n",
    "\n",
    "    3b. shape [6, 1]\n",
    "\n",
    "    3c. shape [3, 2]\n",
    "\n",
    "4. create new array Z of similar shape to X and:\n",
    "\n",
    "    4a. add X to Z\n",
    "\n",
    "    4b. add Z to X\n",
    "\n",
    "    4c. subtract Z from X \n",
    "\n",
    "    4d. reshape X and try to add or subtract from Z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "examples: np.bool | np.int32 | np.float32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "exercises:\n",
    "1. create matrix with random INTEGER numbers ranging from 1 to 100 \n",
    "(try to use function.astype() in this exercise)\n",
    "\n",
    "2. create new matrix with:\n",
    "- numbers ranging from 10 to 20, \n",
    "- shape [2, 5]\n",
    "- values should have type int32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3-dimensional arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "exercise:\n",
    "1. create 3 dimensional array with dimensions [5, 4, 3]\n",
    "\n",
    "2. reshape it to 2-dimensional matrix by flattening\n",
    "   second and third dimention so that your new \n",
    "   shape is [5, 12]\n",
    "   (there are 2 ways to do that, can you do both?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Usefull matrix functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.zeros((2,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.zeros_like(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.ones()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.ones_like()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.arange(12.0).reshape((4, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.rand() \n",
    "# exercise:\n",
    "# 1. create matrix with random values with:\n",
    "# 1a. shape [10, 5]\n",
    "# (notice what values it contains)\n",
    "\n",
    "# 1b. shape [3, 4, 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.sum()\n",
    "# exercise:\n",
    "# 1. sum all elements in matrix\n",
    "\n",
    "# 2. sum X over first axis\n",
    "\n",
    "# 3. sum X over second axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.mean()\n",
    "# exercise:\n",
    "# 1. find mean of all elements in matrix\n",
    "\n",
    "# 2. mean of X over first axis\n",
    "\n",
    "# 3. mean of X over second axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.concatenate()\n",
    "# exercises:\n",
    "# 1. add matrix to matrix in first axis\n",
    "\n",
    "# 2. add matrix to matrix in second axis\n",
    "\n",
    "# 3. add "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.argmax()\n",
    "# exercises:\n",
    "# 1. find indexes with highest value in first axis\n",
    "\n",
    "# 2. find indexes with highest value in second axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.nan_to_num()\n",
    "# exercises:\n",
    "# 1. given matrix A\n",
    "# A = np.array([3, np.NaN, 6])\n",
    "# remove Nan values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interact with pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df=pd.DataFrame(np.random.rand(50,3),columns=[\"a\",\"b\",\"c\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there is a superb way of randomly seperate 20 percent of data (a frequent practice in machine learning to draw some validation data randomly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(df) # The total rows of a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ran=np.random.rand(len(df)) # will generate that many random numbers with value evenly distributed between 0~1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mapping=ran>.8 # return an array telling if each element of ran is bigger than 0.8\n",
    "print(mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df[mapping]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ~ give you the reverse of the True False mapping\n",
    "df[~mapping]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Forward Passing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now I think you are ready to comprehend the most poplular technology idea on this friggin' planet with the following codes. The famous NEURAL NETWORKS. And you are goint to understand basics in 5 minutes!!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine you are doing image recognition, you have dataset MNIST, read some of the MNIST tutorial page on [tensorflow official site](https://www.tensorflow.org/get_started/mnist/beginners)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='mnist_matrix.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Images can be break down to an array of number. MNIST dataset is like the \"hello world\" in machine learning. Some study institution collected hand written digits from 0 ~ 9, tens of thousands of them. You can train machines to understand pictures. You will do MNIST in very recent future, later lessons and achieve 99.7% accuracy.(Many university professors with traditional ML skills are well bellow that accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above picture is 28 by 28, 784 numbers in total (pixels). \n",
    "\n",
    "Let's simplified things a little bit, there are 10 possible categories in this case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "possible_answers=list(range(10))\n",
    "print(\"Possible Answers:\",possible_answers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the convenience of calculation we represent the anwswer of categorical problem as onehot encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0：[1,0,0,0,0,0,0,0,0,0]\n",
    "\n",
    "2：[0,0,1,0,0,0,0,0,0,0]\n",
    "\n",
    "9：[0,0,0,0,0,0,0,0,0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The answer/output of the model will have 10 elements, each represent the possibility of each category. The input is a matrix of 28 by 28, or a vector of length 784 if we flatten the 2d array into 1d.\n",
    "\n",
    "Each of the pixel can range from 0 to 1, the significance of each pixel can decide the possibility of being each number.\n",
    "\n",
    "The following picture dipicts in blue area, the bigger number the pixel got, the higher possibility of being such digit, the red area shows the unlikeliness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='softmax-weights.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, maybe the 4th pixel on the 4th row represent a final possibility of\n",
    "\n",
    "array([  1.55033764e-04,   4.21425463e-04,   1.14555318e-03,\n",
    "         6.94813126e-04,   1.70015166e-01,   4.65747166e-04,\n",
    "         5.14730223e-04,   7.61955112e-01,   6.25450844e-02,\n",
    "         2.08733398e-03])\n",
    "         \n",
    "10 numbers that sums up to 1(100%), means there are 0.000155% possibility the digit is '0', 17% the digit is \"4\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's pretend the following pic tells a written number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_input=np.random.rand(28,28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flatten the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_input=data_input.reshape(784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Design a model structure (The model perameters start with random numbers) :\n",
    "\n",
    "1. The first hidden layer has out put number of 200\n",
    "2. The 2nd hidden layer has output number of 150\n",
    "3. The output layer has output number of 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l1=np.random.rand(784,200)\n",
    "l2=np.random.rand(200,150)\n",
    "l3=np.random.rand(150,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    \"\"\"\n",
    "    Soft max is a function that can make sure the sum of posibility distribution to 1\n",
    "    \"\"\"\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n",
    "# example\n",
    "P=softmax(np.array([1,1,2]))\n",
    "print(P)\n",
    "print(P.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l1out=data_input@l1\n",
    "l2out=l1out@l2\n",
    "finalout=softmax(l2out@l3)\n",
    "print(finalout)\n",
    "\n",
    "# the biggest of the possibility number is the prediction\n",
    "prediction=finalout.argmax(axis=0)\n",
    "print(\"The prediction is digit\",prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "of course, for now the input is fictional, and parameters l1,l2,l3 are also fictional.\n",
    "\n",
    "But the model structure is a forward pass of neural network.\n",
    "\n",
    "We learnt how to get a,b of Y=aX+b from a set of Xs and Ys at school, a,b are the parameters.\n",
    "\n",
    "In this neural network, 784 numbers are input, 10 numbers are output, things like l1, l2, l3 are the parameters. Only there might be billions of parameter numbers in many neural networks. We can use something called stochastic gradient descent (SGD) to get all of them. In some other time you will learn that method, actually it's simpler than mean least square...\n",
    "\n",
    "And you will learn other activation function besides softmax, like relu, sigmoid, also in some other time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='nn.png'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}