t81_558_class_13_05_tpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LSIM-PITWYFa"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_05_tpu.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDTXd8-Lmp8Q"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 13: Advanced/Other Topics**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ncNrAEpzmp8S"
   },
   "source": [
    "# Module 13 Video Material\n",
    "\n",
    "* Part 13.1: Flask and Deep Learning Web Services [[Video]](https://www.youtube.com/watch?v=H73m9XvKHug&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_01_flask.ipynb)\n",
    "* Part 13.2: Interrupting and Continuing Training  [[Video]](https://www.youtube.com/watch?v=kaQCdv46OBA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_02_checkpoint.ipynb)\n",
    "* Part 13.3: Using a Keras Deep Neural Network with a Web Application  [[Video]](https://www.youtube.com/watch?v=OBbw0e-UroI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_03_web.ipynb)\n",
    "* Part 13.4: When to Retrain Your Neural Network [[Video]](https://www.youtube.com/watch?v=K2Tjdx_1v9g&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_04_retrain.ipynb)\n",
    "* **Part 13.5: Tensor Processing Units (TPUs)**  [[Video]](https://www.youtube.com/watch?v=Ygyf3NUqvSc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_13_05_tpu.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lux_6KOXMU94"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fU9UhAxTmp8S",
    "outputId": "8a0287ac-a84c-41c0-c9c1-a8a51600370d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "# Detect Colab if present\n",
    "try:\n",
    "    from google.colab import drive\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TX_m2PyCWn3e"
   },
   "source": [
    "To use Tensor Processing Units (TPUs), you must grant access to Google Cloud Platform (GCP) drives.  If this access is not successfully, you will likely see this error:\n",
    "\n",
    "```\n",
    "InvalidArgumentError: Unable to parse tensor proto\n",
    "```\n",
    "\n",
    "From Google CoLab, issue the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Xomyq-3zQAaz"
   },
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q09yMGGcmp9N"
   },
   "source": [
    "# Part 13.5: Tensor Processing Units (TPUs)\n",
    "\n",
    "This book focuses primarily on NVIDIA Graphics Processing Units (GPUs) for deep learning acceleration. NVIDIA GPUs are not the only option for deep learning acceleration. TensorFlow continues to gain additional support for AMD and Intel GPUs. TPUs are also available from Google cloud platforms to accelerate deep learning. The focus of this book and course is on NVIDIA GPUs because of their wide availability on both local and cloud systems.\n",
    "\n",
    "Though this book focuses on NVIDIA GPUs, we will briefly examine Google Tensor Processing Units (TPUs). These devices are an AI accelerator Application-Specific Integrated Circuit (ASIC) developed by Google. They were designed specifically for neural network machine learning, mainly using Google's TensorFlow software. Google began using TPUs internally in 2015 and in 2018 made them available for third-party use, both as part of its cloud infrastructure and by offering a smaller version of the chip for sale.\n",
    "\n",
    "The full use of a TPU is a complex topic that I only introduced in this part. Supporting TPUs is slightly more complicated than GPUs because specialized coding is needed. Changes are rarely required to adapt CPU code to GPU for most relatively simple mainstream GPU tasks in TensorFlow. I will cover the mild code changes needed to utilize in this part. \n",
    "\n",
    "We will create a regression neural network to count paper clips in this part. I demonstrated this dataset and task several times previously in this book. This part focuses on the utilization of TPUs and not the creation of neural networks. I covered the design of computer vision previously in this book.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c8ixjIi5p8Uy",
    "outputId": "f82eed1a-b2d1-4d84-edf0-13c7b3650e2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-03-13 17:29:19--  https://github.com/jeffheaton/data-mirror/releases/download/v1/paperclips.zip\n",
      "Resolving github.com (github.com)... 140.82.112.3\n",
      "Connecting to github.com (github.com)|140.82.112.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220313%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220313T172920Z&X-Amz-Expires=300&X-Amz-Signature=c08a2d9adeff4affd60042247cd7b0f98d9acb45c89cbf137eadd5d3fe1ccac0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream [following]\n",
      "--2022-03-13 17:29:20--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/408419764/25830812-b9e6-4ddf-93b6-7932d9ef5982?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220313%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220313T172920Z&X-Amz-Expires=300&X-Amz-Signature=c08a2d9adeff4affd60042247cd7b0f98d9acb45c89cbf137eadd5d3fe1ccac0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=408419764&response-content-disposition=attachment%3B%20filename%3Dpaperclips.zip&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 163590691 (156M) [application/octet-stream]\n",
      "Saving to: ‘/content/paperclips.zip’\n",
      "\n",
      "/content/paperclips 100%[===================>] 156.01M   211MB/s    in 0.7s    \n",
      "\n",
      "2022-03-13 17:29:21 (211 MB/s) - ‘/content/paperclips.zip’ saved [163590691/163590691]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "URL = \"https://github.com/jeffheaton/data-mirror/\"\n",
    "DOWNLOAD_SOURCE = URL+\"releases/download/v1/paperclips.zip\"\n",
    "DOWNLOAD_NAME = DOWNLOAD_SOURCE[DOWNLOAD_SOURCE.rfind('/')+1:]\n",
    "\n",
    "if COLAB:\n",
    "  PATH = \"/content\"\n",
    "else:\n",
    "  # I used this locally on my machine, you may need different\n",
    "  PATH = \"/Users/jeff/temp\"\n",
    "\n",
    "EXTRACT_TARGET = os.path.join(PATH,\"clips\")\n",
    "SOURCE = os.path.join(EXTRACT_TARGET, \"paperclips\")\n",
    "\n",
    "# Download paperclip data\n",
    "!wget -O {os.path.join(PATH,DOWNLOAD_NAME)} {DOWNLOAD_SOURCE}\n",
    "!mkdir -p {SOURCE}\n",
    "!mkdir -p {TARGET}\n",
    "!mkdir -p {EXTRACT_TARGET}\n",
    "!unzip -o -j -d {SOURCE} {os.path.join(PATH, DOWNLOAD_NAME)} >/dev/null\n",
    "\n",
    "# Add filenames\n",
    "df_train = pd.read_csv(os.path.join(SOURCE, \"train.csv\"))\n",
    "df_train['filename'] = \"clips-\" + df_train.id.astype(str) + \".jpg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pw_ysSVRiZcP"
   },
   "source": [
    "## Preparing Data for TPUs\n",
    "\n",
    "To present the paperclips dataset to the TPU, we will convert the images to a Keras Dataset. Because we will load the entire dataset to RAM, we will only utilize the first 1,000 images. We previously loaded the labels from the **train.csv** file. The following code loads these images and converts them to a Keras dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RY4NU-vz_Eis",
    "outputId": "586c3d29-080d-4d38-cbe7-3b25bac78963"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [00:02<00:00, 413.74it/s]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import keras_preprocessing\n",
    "import glob, os\n",
    "import tqdm\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "\n",
    "IMG_SHAPE = (128,128)\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "# Resize each image and convert the 0-255 ranged RGB values to 0-1 range.\n",
    "def load_images(files, img_shape):\n",
    "  cnt = len(files)\n",
    "  x = np.zeros((cnt,)+img_shape+(3,),dtype=np.float32)\n",
    "  i = 0\n",
    "  for file in tqdm.tqdm(files):\n",
    "    img = Image.open(file)\n",
    "    img = img.resize(img_shape)\n",
    "    img = np.array(img)\n",
    "    img = img/255\n",
    "    x[i,:,:,:] = img\n",
    "    i+=1\n",
    "  return x\n",
    "\n",
    "# Process training data \n",
    "df_train = pd.read_csv(os.path.join(SOURCE, \"train.csv\"))\n",
    "df_train['filename'] = \"clips-\" + df_train.id.astype(str) + \".jpg\"\n",
    "\n",
    "# Use only the first 1000 images\n",
    "df_train = df_train[0:1000]\n",
    "\n",
    "# Load images\n",
    "images = [os.path.join(SOURCE,x) for x in df_train.filename]\n",
    "x = load_images(images, IMG_SHAPE)\n",
    "y = df_train.clip_count.values\n",
    "\n",
    "# Convert to dataset\n",
    "dataset = tf.data.Dataset.from_tensor_slices((x, y))\n",
    "dataset = dataset.batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ig6mQ4tBlDsp"
   },
   "source": [
    "TPUs are typically Cloud TPU workers, different from the local process running the user's Python program. Thus, it would be best to do some initialization work to connect to the remote cluster and initialize the TPUs. The TPU argument to **tf.distribute.cluster_resolver**. **TPUClusterResolver** is a unique address just for Colab. If you are running your code on Google Compute Engine (GCE), you should instead pass in the name of your Cloud TPU. The following code performs this initialization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7O04z13LV0Dp",
    "outputId": "f405a27a-562a-4441-ce9c-6cf752686025"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Deallocate tpu buffers before initializing tpu system.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Deallocate tpu buffers before initializing tpu system.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Initializing the TPU system: grpc://10.97.246.186:8470\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Initializing the TPU system: grpc://10.97.246.186:8470\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Finished initializing TPU system.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Finished initializing TPU system.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: grpc://10.97.246.186:8470\n",
      "INFO:tensorflow:Found TPU system:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Found TPU system:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Cores: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Cores: 8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Workers: 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Workers: 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of replicas: 8\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "try:\n",
    "    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect()\n",
    "    print(\"Device:\", tpu.master())\n",
    "    strategy = tf.distribute.TPUStrategy(tpu)\n",
    "except:\n",
    "    strategy = tf.distribute.get_strategy()\n",
    "print(\"Number of replicas:\", strategy.num_replicas_in_sync)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8rVZJBMIgqJq"
   },
   "source": [
    "We will now use a ResNet neural network as a basis for our neural network. We begin by loading, from Keras, the ResNet50 network. We will redefine both the input shape and output of the ResNet model, so we will not transfer the weights. Since we redefine the input, the weights are of minimal value. We specify **include_top** as False because we will change the input resolution. We also specify **weights** as false because we must retrain the network after changing the top input layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6MJpmLyhtahJ",
    "outputId": "f92c5133-c710-498f-df5a-1614e7be7cbe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "32/32 [==============================] - 91s 1s/step - loss: 412.2361 - rmse: 20.3036\n",
      "Epoch 2/100\n",
      "32/32 [==============================] - 1s 43ms/step - loss: 156.3177 - rmse: 12.5027\n",
      "Epoch 3/100\n",
      "32/32 [==============================] - 1s 43ms/step - loss: 126.0154 - rmse: 11.2257\n",
      "Epoch 4/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 81.0153 - rmse: 9.0008\n",
      "Epoch 5/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 56.0450 - rmse: 7.4863\n",
      "Epoch 6/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 54.7298 - rmse: 7.3980\n",
      "Epoch 7/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 51.7157 - rmse: 7.1914\n",
      "Epoch 8/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 77.8686 - rmse: 8.8243\n",
      "Epoch 9/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 49.4168 - rmse: 7.0297\n",
      "Epoch 10/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 54.6380 - rmse: 7.3918\n",
      "Epoch 11/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 53.8622 - rmse: 7.3391\n",
      "Epoch 12/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 42.4151 - rmse: 6.5127\n",
      "Epoch 13/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 67.3325 - rmse: 8.2056\n",
      "Epoch 14/100\n",
      "32/32 [==============================] - 2s 46ms/step - loss: 113.7659 - rmse: 10.6661\n",
      "Epoch 15/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 78.4862 - rmse: 8.8592\n",
      "Epoch 16/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 76.3148 - rmse: 8.7358\n",
      "Epoch 17/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 53.6310 - rmse: 7.3233\n",
      "Epoch 18/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 35.9240 - rmse: 5.9937\n",
      "Epoch 19/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 28.7515 - rmse: 5.3620\n",
      "Epoch 20/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 24.2422 - rmse: 4.9236\n",
      "Epoch 21/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 34.2141 - rmse: 5.8493\n",
      "Epoch 22/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 25.2991 - rmse: 5.0298\n",
      "Epoch 23/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 15.2113 - rmse: 3.9002\n",
      "Epoch 24/100\n",
      "32/32 [==============================] - 1s 43ms/step - loss: 16.7044 - rmse: 4.0871\n",
      "Epoch 25/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 17.5892 - rmse: 4.1940\n",
      "Epoch 26/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 18.7181 - rmse: 4.3264\n",
      "Epoch 27/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 16.7338 - rmse: 4.0907\n",
      "Epoch 28/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 15.6866 - rmse: 3.9606\n",
      "Epoch 29/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 26.5499 - rmse: 5.1527\n",
      "Epoch 30/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 22.8419 - rmse: 4.7793\n",
      "Epoch 31/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 32.6161 - rmse: 5.7111\n",
      "Epoch 32/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 45.6118 - rmse: 6.7537\n",
      "Epoch 33/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 13.8262 - rmse: 3.7184\n",
      "Epoch 34/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 13.2402 - rmse: 3.6387\n",
      "Epoch 35/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 25.0043 - rmse: 5.0004\n",
      "Epoch 36/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 16.8132 - rmse: 4.1004\n",
      "Epoch 37/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 16.7914 - rmse: 4.0977\n",
      "Epoch 38/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 44.0505 - rmse: 6.6371\n",
      "Epoch 39/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 37.8859 - rmse: 6.1552\n",
      "Epoch 40/100\n",
      "32/32 [==============================] - 1s 43ms/step - loss: 45.2483 - rmse: 6.7267\n",
      "Epoch 41/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 40.9563 - rmse: 6.3997\n",
      "Epoch 42/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 29.2618 - rmse: 5.4094\n",
      "Epoch 43/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 27.8391 - rmse: 5.2763\n",
      "Epoch 44/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 74.5348 - rmse: 8.6334\n",
      "Epoch 45/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 67.7960 - rmse: 8.2338\n",
      "Epoch 46/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 92.6046 - rmse: 9.6231\n",
      "Epoch 47/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 84.2020 - rmse: 9.1762\n",
      "Epoch 48/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 85.4522 - rmse: 9.2440\n",
      "Epoch 49/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 71.1020 - rmse: 8.4322\n",
      "Epoch 50/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 108.8298 - rmse: 10.4321\n",
      "Epoch 51/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 79.8435 - rmse: 8.9355\n",
      "Epoch 52/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 55.1271 - rmse: 7.4248\n",
      "Epoch 53/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 45.2738 - rmse: 6.7286\n",
      "Epoch 54/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 40.6076 - rmse: 6.3724\n",
      "Epoch 55/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 49.2865 - rmse: 7.0204\n",
      "Epoch 56/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 52.1646 - rmse: 7.2225\n",
      "Epoch 57/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 30.5701 - rmse: 5.5290\n",
      "Epoch 58/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 31.0364 - rmse: 5.5710\n",
      "Epoch 59/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 25.4160 - rmse: 5.0414\n",
      "Epoch 60/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 22.1702 - rmse: 4.7085\n",
      "Epoch 61/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 24.5868 - rmse: 4.9585\n",
      "Epoch 62/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 18.4491 - rmse: 4.2952\n",
      "Epoch 63/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 25.1780 - rmse: 5.0178\n",
      "Epoch 64/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 14.7577 - rmse: 3.8416\n",
      "Epoch 65/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 15.3932 - rmse: 3.9234\n",
      "Epoch 66/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 22.6359 - rmse: 4.7577\n",
      "Epoch 67/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 13.0876 - rmse: 3.6177\n",
      "Epoch 68/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 19.2264 - rmse: 4.3848\n",
      "Epoch 69/100\n",
      "32/32 [==============================] - 1s 43ms/step - loss: 15.3374 - rmse: 3.9163\n",
      "Epoch 70/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 14.3568 - rmse: 3.7890\n",
      "Epoch 71/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 19.8507 - rmse: 4.4554\n",
      "Epoch 72/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 13.0564 - rmse: 3.6134\n",
      "Epoch 73/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 14.6215 - rmse: 3.8238\n",
      "Epoch 74/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 16.7923 - rmse: 4.0978\n",
      "Epoch 75/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 23.6171 - rmse: 4.8597\n",
      "Epoch 76/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 20.1067 - rmse: 4.4840\n",
      "Epoch 77/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 23.4237 - rmse: 4.8398\n",
      "Epoch 78/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 25.5370 - rmse: 5.0534\n",
      "Epoch 79/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 33.7498 - rmse: 5.8095\n",
      "Epoch 80/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 22.2081 - rmse: 4.7125\n",
      "Epoch 81/100\n",
      "32/32 [==============================] - 1s 46ms/step - loss: 34.0196 - rmse: 5.8326\n",
      "Epoch 82/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 25.7848 - rmse: 5.0779\n",
      "Epoch 83/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 23.4041 - rmse: 4.8378\n",
      "Epoch 84/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 23.9674 - rmse: 4.8956\n",
      "Epoch 85/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 29.0349 - rmse: 5.3884\n",
      "Epoch 86/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 21.8591 - rmse: 4.6754\n",
      "Epoch 87/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 14.2806 - rmse: 3.7790\n",
      "Epoch 88/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 18.8377 - rmse: 4.3402\n",
      "Epoch 89/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 13.1473 - rmse: 3.6259\n",
      "Epoch 90/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 9.5658 - rmse: 3.0929\n",
      "Epoch 91/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 19.8885 - rmse: 4.4597\n",
      "Epoch 92/100\n",
      "32/32 [==============================] - 1s 45ms/step - loss: 14.8018 - rmse: 3.8473\n",
      "Epoch 93/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 7.1501 - rmse: 2.6740\n",
      "Epoch 94/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 18.9983 - rmse: 4.3587\n",
      "Epoch 95/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 14.1897 - rmse: 3.7669\n",
      "Epoch 96/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 11.6128 - rmse: 3.4078\n",
      "Epoch 97/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 21.4982 - rmse: 4.6366\n",
      "Epoch 98/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 23.1815 - rmse: 4.8147\n",
      "Epoch 99/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 18.3960 - rmse: 4.2891\n",
      "Epoch 100/100\n",
      "32/32 [==============================] - 1s 44ms/step - loss: 10.4749 - rmse: 3.2365\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.applications.resnet50 import ResNet50\n",
    "from tensorflow.keras.layers import Input\n",
    "from tensorflow.keras.layers import Dense, GlobalAveragePooling2D\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from tensorflow.keras.metrics import RootMeanSquaredError\n",
    "\n",
    "def create_model():\n",
    "  input_tensor = Input(shape=IMG_SHAPE+(3,))\n",
    "\n",
    "  base_model = ResNet50(\n",
    "      include_top=False, weights=None, input_tensor=input_tensor,\n",
    "      input_shape=None)\n",
    "\n",
    "  x=base_model.output\n",
    "  x=GlobalAveragePooling2D()(x)\n",
    "  x=Dense(1024,activation='relu')(x) \n",
    "  x=Dense(1024,activation='relu')(x) \n",
    "  model=Model(inputs=base_model.input,outputs=Dense(1)(x))\n",
    "  return model\n",
    "\n",
    "with strategy.scope():\n",
    "  model = create_model()\n",
    "\n",
    "  model.compile(loss = 'mean_squared_error', optimizer='adam', \n",
    "              metrics=[RootMeanSquaredError(name=\"rmse\")])\n",
    "\n",
    "  history = model.fit(dataset, epochs=100, steps_per_epoch=32, \n",
    "                    verbose = 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9v9hQ59MUwI2"
   },
   "source": [
    "You might receive the following error while fitting the neural network.\n",
    "\n",
    "```\n",
    "InvalidArgumentError: Unable to parse tensor proto\n",
    "```\n",
    "\n",
    "If you do receive this error, it is likely because you are missing proper authentication to access Google Drive to store your datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "base-tpu.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}