1_MNIST_MLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  神經網路(Neural  Network)是由人類的大腦神經結構的運作借鏡而來，在機器學習的世界中，神經元就像是大腦的神經細胞，是神經網路最基礎的結構，在它們相互結合下，建構整個龐大的運作網路，實現學習、處理及預測等功能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OYVsBH41KS-T"
   },
   "source": [
    "# 使用多層感知器Multilayer perceptron ( MLP )\n",
    "---\n",
    "* 多層感知器就是神經網路\n",
    "![MLP示意圖](MLP.jpeg)\n",
    "\n",
    "\n",
    "神經元在接收輸入訊號後可以想像它是儲存了一個數字的容器，其值介於0到1之間。\n",
    "\n",
    "以 $28 \\times 28$ 像素的手寫辨識圖片來說，每個像素就是一個神經元，也就是一張圖片在輸入層總共有784個神經元，每個神經元都儲存了一個數字來代表對應像素的灰階值，數值的範圍介於0跟1之間。\n",
    "\n",
    "而灰階值0代表黑色，1代表白色，這些數字我們稱為激勵值，數值越大則該神經元就越亮。在輸入時要將矩陣平面化(將28列前後相接成一列)，也就是這784個神經元組成了神經網路的第一層。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "79OVv5yfMG2B"
   },
   "source": [
    "### Mnist資料集共有 60000 筆訓練資料，將訓練資料的  Feature(數字圖片特徵值) 和 Label(數字真值實) 都先經過預處理，作為多層感知器的輸入、輸出，然後進行模型訓練。\n",
    "\n",
    "![MLP步驟](MLP_step.jpg)\n",
    "\n",
    "\n",
    "### 模型訓練完成以後就可以用來作預測，將要預測的數字圖片，先經過預處理變成Feature(數字圖片特徵值)，就可送給模型作預測，得到 0~9 數字的預測結果。\n",
    "\n",
    "## MLP 建立步驟\n",
    "\n",
    "1. 資料處理\n",
    "2. 建立模型\n",
    "3. 訓練模型 \n",
    "4. 評估模型準確度\n",
    "5. 進行預測"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "# 指定亂數種子\n",
    "seed = 7\n",
    "np.random.seed(seed)\n",
    "\n",
    "# 載入資料集\n",
    "(X_train, Y_train), (X_test, Y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 資料預處理\n",
    "\n",
    "* Features(數字影像特徵值)資料預處理: 28*28數字影像 ➔ 784 , type:float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 轉換成 2**28 = 784 的向量\n",
    "X_train_list = X_train.reshape(X_train.shape[0], 28*28).astype(\"float32\")\n",
    "X_test_list = X_test.reshape(X_test.shape[0], 28*28).astype(\"float32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*TEST: 可輸入  X_train.shape 查看 X_train的張量大小*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*TEST: 可輸入  X_train_list.shape 查看 X_train_list的張量大小*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Labels(數字影像真實值)資料預處理: 數字影像Image的數字影像標準化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 因為是固定範圍, 所以執行標準化, 從 0-255 至 0-1\n",
    "X_train_normalize = X_train_list / 255\n",
    "X_test_normalize  = X_test_list / 255\n",
    "\n",
    "Y_test_bk = Y_test.copy() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### labels資料預處理\n",
    "\n",
    "* label原本是0~9的數字  ➔  10個0或1的組合\n",
    "* 數字3經過 one-hot encoding ➔ 0001000000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One-hot編碼 (輸出對應)\n",
    "Y_train_OneHot = to_categorical(Y_train)\n",
    "Y_test_OneHot = to_categorical(Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*TEST: 查看結果*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (Y_train[0] )\n",
    "print (Y_train_OneHot[0] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 啟動函數 Activation Function\n",
    "\n",
    "* Step function\n",
    "* Sigmoid function \n",
    "* ReLu (Rectified Linear Unit) function\n",
    "* Tanh function\n",
    "* Softmax function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1A_T1aYvKav7"
   },
   "source": [
    "# 2.建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 建立 Sequential 順序模組\n",
    "model = Sequential()\n",
    "\n",
    "\n",
    "# 模型加入【輸入層】與【隱藏層】\n",
    "model.add( Dense( input_dim = 28*28             # 輸入層有 28*28=784 個神經元\n",
    "                , units = 256                   # 隱藏層有 256 個神經元    (值越大, 訓練越精準, 相對訓練時間也越久)\n",
    "                , kernel_initializer = 'normal' # 使用 normal 初始化 weight 權重與 bias 偏差值\n",
    "                , activation = 'relu'           # 使用 relu 啟動函數\n",
    "                ))\n",
    "\n",
    "\n",
    "# 模型加入【輸出層】\n",
    "model.add( Dense( units = 10                    # 輸出層有 10 個神經元 (因為數字只有 0 ~ 9)\n",
    "                , kernel_initializer = 'normal' # 使用 normal 初始化 weight 權重與 bias 偏差值\n",
    "                , activation = 'softmax'        # 使用 softmax 啟動函函數 (softmax 值越高, 代表機率越大)\n",
    "                ))\n",
    "\n",
    "# 顯示模型摘要資訊\n",
    "print (model.summary() )   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "para # : 784*256+256 = 200960\n",
    "\n",
    "* $h_1=relu( X \\times W_1 +b_1 )$ \n",
    "\n",
    "para # : 256*10+ 10 = 2570\n",
    "\n",
    "* $y=softmax( h_2 \\times W_2 +b_2 )$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 損失函數 loss Function\n",
    "\n",
    "* 均方誤差 mean square error\n",
    "* 交叉熵 Croee-Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6E0pdWOxKn4H"
   },
   "source": [
    "# 3. Model 模型進行【Train 訓練】"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 設定模型的訓練方式 ##\n",
    "\n",
    "model.compile( loss='categorical_crossentropy' # 設定 Loss 損失函數 為 categorical_crossentropy 使用交叉熵\n",
    "             , optimizer = 'adam'              # 設定 Optimizer 最佳化方法 為 adam\n",
    "             , metrics = ['accuracy']          # 設定 Model 評估準確率方法 為 accuracy\n",
    "             )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 訓練模型\n",
    "history = model.fit(               # 訓練的歷史記錄, 會會回傳到指定變數 history\n",
    "          x = X_train_normalize    # 設定 圖片 Features 特徵值 (mnist 提供 60000 筆資料)\n",
    "        , y = Y_train_OneHot       # 設定 圖片 Label    真實值 (mnist 提供 60000 筆資料)\n",
    "        , validation_split = 0.2   # 設定 有多少筆驗證         (60000*0.2=12000 筆驗證, 60000*0.8=48000 筆訓練)\n",
    "        , epochs = 10              # 設定 訓練次數             (值 10 以上,  值越大, 訓練時間越久, 但訓練越精準)\n",
    "        , batch_size = 128         # 設定 訓練時每批次有多少筆 (值 100 以上, 值越大, 訓練速度越快, 但需記憶體要夠大)\n",
    "        , verbose = 2              # 是否 顯示訓練過程         (0: 不顯示, 1: 詳細顯示, 2: 簡易顯示)\n",
    ")\n",
    "\n",
    "# 執行的顯示結果 (這會花一些時間, 然後會逐次顯示訓練結果)\n",
    "# loss:     使用訓練資料, 得到的損失函數誤差值 (值越小, 代表準確率越高)\n",
    "# acc:      使用訓練資料, 得到的評估準確率    (值在 0~1, 值越大, 代表準確率越高)\n",
    "# val_loss: 使用驗證資料, 得到的損失函數誤差值 (值越小, 代表準確率越高)\n",
    "# val_acc:  使用驗證資料, 得到的評估準確率    (值在 0~1, 值越大, 代表準確率越高)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0RB4jtxBKw7w"
   },
   "source": [
    "#  4. 評估模型準確度\n",
    "**顯示圖表來分析模型的訓練過程** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 顯示圖表來分析模型的訓練過程\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# 顯示訓練和驗證損失\n",
    "loss = history.history[\"loss\"]\n",
    "epochs = range(1, len(loss)+1)\n",
    "val_loss = history.history[\"val_loss\"]\n",
    "\n",
    "plt.plot(epochs, loss, \"bo-\", label=\"Training Loss\")\n",
    "plt.plot(epochs, val_loss, \"ro--\", label=\"Validation Loss\")\n",
    "plt.title(\"Training and Validation Loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "# 顯示訓練和驗證準確度\n",
    "acc = history.history['acc']\n",
    "epochs = range(1, len(acc)+1)\n",
    "val_acc = history.history['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, \"bo-\", label=\"Training Acc\")\n",
    "plt.plot(epochs, val_acc, \"ro--\", label=\"Validation Acc\")\n",
    "plt.title(\"Training and Validation Accuracy\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 儲存模型\n",
    "model.save('mnist_mlp_h256.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**overfitting 過度擬合**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EjQFDCwVQwR1"
   },
   "source": [
    "# 5. 進行預測"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 評估模型準確度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sores = model.evaluate( X_test_normalize , Y_test_OneHot)\n",
    "print ()\n",
    "print ( 'accuracy=',sores[1] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ph4n-VCRz1Go"
   },
   "source": [
    "* 預測結果與原始圖形"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 執行預測\n",
    "Y_pred = model.predict_classes(X_test_normalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import  matplotlib.pyplot as plt\n",
    "fig = plt.figure(figsize=(10,10))\n",
    "\n",
    "for i in range(20):\n",
    "  ax = plt.subplot(4,5,i+1)\n",
    "  ax.imshow(X_test[i] ,cmap='gray')\n",
    "  ax.set_title('Label:'+str(Y_test[i])+ '  pre:'+str(Y_pred[i]) ,fontsize= 11      )\n",
    "  ax.axis('off')\n",
    "plt.subplots_adjust( hspace=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HkT3Z_5yLK-p"
   },
   "source": [
    "### 計算分類的預測值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TD5hUZiuY_Uu"
   },
   "source": [
    "**混淆矩陣**\n",
    "* 混淆矩陣(Confusion matrix)是一種對分類模型進行效果評估的方法\n",
    "* 通過將模型預測的數據與測試數據進行對比，使用準確率，覆蓋率和命中率等指標對模型的分類效果進行度量。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 計算分類的預測值\n",
    "print(\"\\nPredicting ...\")\n",
    "Y_pred = model.predict_classes(X_test_normalize)\n",
    "\n",
    "\n",
    "import pandas as pd\n",
    "# 顯示混淆矩陣\n",
    "tb = pd.crosstab(Y_test_bk.astype(int), Y_pred.astype(int),\n",
    "                 rownames=[\"label\"], colnames=[\"predict\"])\n",
    "print(tb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MbZzT6wSZCZz"
   },
   "source": [
    "**建立真實值與預測 dataframe**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame( {'label': Y_test_bk.astype(int), 'predict': Y_pred.astype(int) }   )\n",
    "df[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_DXCxIEl0KBc"
   },
   "source": [
    "**匯出0~9數字預測機率**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 7\n",
    "digit = X_test[i].reshape(28, 28)\n",
    "# 將圖片轉換成 4D 張量\n",
    "X_test_digit = X_test[i].reshape(1, 28, 28, 1).astype(\"float32\")\n",
    "\n",
    "print ( '-------')\n",
    "# 因為是固定範圍, 所以執行正規化, 從 0-255 至 0-1\n",
    "X_test_digit = X_test_digit / 255\n",
    "\n",
    "# 繪出圖表的預測結果\n",
    "plt.figure()\n",
    "plt.subplot(1,2,1)\n",
    "plt.title(\"Example of Digit:\" + str(Y_test[i]))\n",
    "plt.imshow(digit, cmap=\"gray\")\n",
    "plt.axis(\"off\")\n",
    "\n",
    "# 預測結果的機率\n",
    "print(\"Predicting ...\")\n",
    "probs = model.predict_proba(X_test_normalize)\n",
    "plt.subplot(1,2,2)\n",
    "print (probs.shape)\n",
    "plt.title(\"Probabilities for Each Digit Class\")\n",
    "plt.bar(np.arange(10), probs[i], align=\"center\")\n",
    "plt.xticks(np.arange(10),np.arange(10).astype(str))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[ ( df.label==5  ) & (df.predict==3)   ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = 340\n",
    "\n",
    "import  matplotlib.pyplot as plt\n",
    "\n",
    "# 繪出圖表的預測結果\n",
    "plt.figure()\n",
    "plt.subplot(1,2,1)\n",
    "\n",
    "plt.imshow(X_test[index] ,cmap='gray')\n",
    "plt.title('Label:'+str(Y_test[index])+ '  pre:'+str(Y_pred[index]) ,fontsize= 12      )\n",
    "plt.axis('off')\n",
    "\n",
    "# 預測結果的機率\n",
    "print(\"Predicting ...\")\n",
    "probs = model.predict_proba(X_test_normalize)\n",
    "plt.subplot(1,2,2)\n",
    "print (probs.shape)\n",
    "plt.title(\"Probabilities for Each Digit Class\")\n",
    "plt.bar(np.arange(10), probs[index], align=\"center\")\n",
    "plt.xticks(np.arange(10),np.arange(10).astype(str))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overfitting問題\n",
    "* Overfitting(過度訓練):當可選擇的參數自由度超過資料所包含的資訊內容時，這會破壞模型一般化的能力。\n",
    "* 解決方法:\n",
    "\n",
    "  (1)增加數據量, 大部分過擬合產生的原因是因為數據量太少了。\n",
    "  \n",
    "  (2)加入DropOut功能，在訓練的時候，我們隨機忽略掉一些神經元和神經元的連結，讓每一次預測結果都不會太過依賴於其中某部分特定神經元。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyMFm1Cg5DivAwKg9xj2gEhw",
   "name": "MNIST_MLP.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}