dgx_benchamrking_tf_mirrored_stratergy.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w7zmWQvkUc1Q"
      },
      "source": [
        "<h2 style='color:blue'>Small Image Classification Using Simple Aritifical Neural Network: GPU Benchmarking</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Ja7edJfUc1U"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"4\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": false,
        "id": "mkRr7nBGUc1V"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "# Version Information\n",
        "# tensorflow 2.2.0 , Cudnn7.6.5 and Cuda 10.1 , python 3.8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8_JSqFIYUc1W"
      },
      "source": [
        "**This command shows list of physical devices available for tensorflow. You can see GPU listed here. If you have NVIDIA GPU you need to install CUDA toolkit and cuDNN as per instruction on this webpage. Without proper installation you will not see GPU in list of devices**\n",
        "\n",
        "https://shawnhymel.com/1961/how-to-install-tensorflow-with-gpu-support-on-windows/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "BmHU4miUUc1W",
        "outputId": "06660177-e882-47ba-de7a-c57cdcea07b6"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),\n",
              " PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),\n",
              " PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'),\n",
              " PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'),\n",
              " PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.config.experimental.list_physical_devices()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WN53_lyEUc1W",
        "outputId": "ded034a6-4cc3-48c3-800b-c3855f33ed8d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'2.4.0'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.__version__"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WyvSqDsnUc1X",
        "outputId": "079f244d-bfb0-4232-b11d-6d3b2f4702f1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tf.test.is_built_with_cuda()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dq-2OnYkUc1X"
      },
      "source": [
        "<h4 style=\"color:purple\">Load the dataset</h4>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fzXEls8iUc1X"
      },
      "source": [
        "Our dataset contains 60000 small training images that belongs to one of the below 10 classes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgmrLhJbUc1X"
      },
      "source": [
        "<img src=\"small_images.jpg\" />"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RngWg86OUc1Y",
        "outputId": "ddce4ab7-e6dc-4dfb-a74d-7c72aa0c1f68"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
            "170500096/170498071 [==============================] - 31s 0us/step\n"
          ]
        }
      ],
      "source": [
        "(X_train, y_train), (X_test,y_test) = tf.keras.datasets.cifar10.load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wmRnCRw7Uc1Y",
        "outputId": "cd894260-b9b0-45b6-f37d-fc9f7c3b49c4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(50000, 32, 32, 3)"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "SLNAfRXeUc1Y",
        "outputId": "678b7dcb-505c-44ae-9f4a-95d786f88686"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(50000, 1)"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_train.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c5haKAKrUc1Y"
      },
      "outputs": [],
      "source": [
        "classes = [\"airplane\",\"automobile\",\"bird\",\"cat\",\"deer\",\"dog\",\"frog\",\"horse\",\"ship\",\"truck\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vj-dGc_7Uc1Y",
        "outputId": "4c300dbb-f85f-4541-be27-30e41d3fe2a4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'deer'"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "classes[y_train[3][0]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "NVpsc_4KUc1Y",
        "outputId": "3664879b-fdd3-456e-a2e7-950ced1e50f8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[6],\n",
              "       [9],\n",
              "       [9]], dtype=uint8)"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_train[:3]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T1I-KH1cUc1Z",
        "outputId": "526b3e27-b831-42c2-d0ff-c68712c5b66b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(10000, 1)"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_test.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "Lm_rkWYpUc1Z",
        "outputId": "e76765f8-778a-4b93-96ad-054d8554fb6c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(50000, 32, 32, 3)"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cwCde1SDUc1Z"
      },
      "source": [
        "<h4 style=\"color:purple\">Preprocessing: Scale images</h4>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "1q73Jd9HUc1Z"
      },
      "outputs": [],
      "source": [
        "X_train_scaled = X_train / 255\n",
        "X_test_scaled = X_test / 255"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-TygNbDZUc1Z"
      },
      "outputs": [],
      "source": [
        "y_train_categorical = keras.utils.to_categorical(\n",
        "    y_train, num_classes=10, dtype='float32'\n",
        ")\n",
        "y_test_categorical = keras.utils.to_categorical(\n",
        "    y_test, num_classes=10, dtype='float32'\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YREx2EfXUc1Z",
        "outputId": "3172eeba-358e-4778-eb8a-c41f6a16b0df"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[6],\n",
              "       [9],\n",
              "       [9],\n",
              "       [4],\n",
              "       [1]], dtype=uint8)"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_train[0:5]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "VVZOyR1wUc1Z",
        "outputId": "cd0e5743-18e2-4bcf-acfb-a1b07ac74fae"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
              "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
              "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_train_categorical[0:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kyny9t_hUc1Z"
      },
      "source": [
        "<h4 style=\"color:purple\">Model building and training</h4>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w0kBJOd0Uc1a"
      },
      "outputs": [],
      "source": [
        "def get_model():\n",
        "    model = keras.Sequential([\n",
        "            keras.layers.Flatten(input_shape=(32,32,3)),\n",
        "            keras.layers.Dense(3000, activation='relu'),\n",
        "            keras.layers.Dense(1000, activation='relu'),\n",
        "            keras.layers.Dense(10, activation='sigmoid')\n",
        "        ])\n",
        "\n",
        "    model.compile(optimizer='SGD',\n",
        "                  loss='categorical_crossentropy',\n",
        "                  metrics=['accuracy'])\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pvxrqRl7Uc1a"
      },
      "outputs": [],
      "source": [
        "train_tf_dataset = tf.data.Dataset.from_tensor_slices((X_train_scaled, y_train_categorical))\n",
        "test_tf_dataset = tf.data.Dataset.from_tensor_slices((X_test_scaled, y_test_categorical))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0eCZ7XRGUc1a",
        "outputId": "6a3f0805-20cb-47be-af28-5bf37ed73abc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')\n"
          ]
        }
      ],
      "source": [
        "strategy = tf.distribute.MirroredStrategy()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pJr3uBDXUc1a",
        "outputId": "b88a0dba-cafe-4447-89ab-1ecea17415ad"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "strategy.num_replicas_in_sync"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iiRaqXQKUc1a"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE_PER_REPLICA = 250\n",
        "BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\n",
        "\n",
        "\n",
        "train_dataset = train_tf_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n",
        "test_dataset = test_tf_dataset.batch(BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": false,
        "id": "s4ddW7muUc1a",
        "outputId": "4a182e87-9efa-43ed-a537-f44cb9b45e20"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "50/50 [==============================] - 3s 9ms/step - loss: 2.1940 - accuracy: 0.2043\n",
            "Epoch 2/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.9763 - accuracy: 0.3150\n",
            "Epoch 3/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.8989 - accuracy: 0.3441\n",
            "Epoch 4/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.8531 - accuracy: 0.3617\n",
            "Epoch 5/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.8211 - accuracy: 0.3709\n",
            "Epoch 6/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.7967 - accuracy: 0.3781\n",
            "Epoch 7/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.7769 - accuracy: 0.3862\n",
            "Epoch 8/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.7599 - accuracy: 0.3928\n",
            "Epoch 9/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.7449 - accuracy: 0.3973\n",
            "Epoch 10/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.7315 - accuracy: 0.4023\n",
            "Epoch 11/50\n",
            "50/50 [==============================] - 0s 10ms/step - loss: 1.7192 - accuracy: 0.4055\n",
            "Epoch 12/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.7077 - accuracy: 0.4094\n",
            "Epoch 13/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.6971 - accuracy: 0.4140\n",
            "Epoch 14/50\n",
            "50/50 [==============================] - 0s 10ms/step - loss: 1.6871 - accuracy: 0.4177\n",
            "Epoch 15/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.6777 - accuracy: 0.4205\n",
            "Epoch 16/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.6687 - accuracy: 0.4232\n",
            "Epoch 17/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.6602 - accuracy: 0.4263\n",
            "Epoch 18/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.6521 - accuracy: 0.4294\n",
            "Epoch 19/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.6443 - accuracy: 0.4321\n",
            "Epoch 20/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.6368 - accuracy: 0.4352\n",
            "Epoch 21/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.6296 - accuracy: 0.4379\n",
            "Epoch 22/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.6227 - accuracy: 0.4402\n",
            "Epoch 23/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.6160 - accuracy: 0.4422\n",
            "Epoch 24/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.6095 - accuracy: 0.4440\n",
            "Epoch 25/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.6032 - accuracy: 0.4464\n",
            "Epoch 26/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5972 - accuracy: 0.4474\n",
            "Epoch 27/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5913 - accuracy: 0.4503\n",
            "Epoch 28/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.5856 - accuracy: 0.4529\n",
            "Epoch 29/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.5801 - accuracy: 0.4547\n",
            "Epoch 30/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5748 - accuracy: 0.4561\n",
            "Epoch 31/50\n",
            "50/50 [==============================] - 0s 10ms/step - loss: 1.5696 - accuracy: 0.4576\n",
            "Epoch 32/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5645 - accuracy: 0.4600\n",
            "Epoch 33/50\n",
            "50/50 [==============================] - 0s 10ms/step - loss: 1.5596 - accuracy: 0.4617\n",
            "Epoch 34/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.5548 - accuracy: 0.4639\n",
            "Epoch 35/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.5502 - accuracy: 0.4657\n",
            "Epoch 36/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5457 - accuracy: 0.4666\n",
            "Epoch 37/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.5413 - accuracy: 0.4684\n",
            "Epoch 38/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.5370 - accuracy: 0.4700\n",
            "Epoch 39/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5328 - accuracy: 0.4711\n",
            "Epoch 40/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.5286 - accuracy: 0.4727\n",
            "Epoch 41/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.5247 - accuracy: 0.4742\n",
            "Epoch 42/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.5207 - accuracy: 0.4754\n",
            "Epoch 43/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.5167 - accuracy: 0.4770\n",
            "Epoch 44/50\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 1.5128 - accuracy: 0.4782\n",
            "Epoch 45/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.5089 - accuracy: 0.4796\n",
            "Epoch 46/50\n",
            "50/50 [==============================] - 0s 9ms/step - loss: 1.5051 - accuracy: 0.4800\n",
            "Epoch 47/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.5012 - accuracy: 0.4814\n",
            "Epoch 48/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.4975 - accuracy: 0.4821\n",
            "Epoch 49/50\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 1.4937 - accuracy: 0.4830\n",
            "Epoch 50/50\n",
            "50/50 [==============================] - 0s 8ms/step - loss: 1.4900 - accuracy: 0.4839\n",
            "47.6 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit -n1 -r1\n",
        "with strategy.scope():\n",
        "    gpu_model = get_model()\n",
        "    gpu_model.fit(train_dataset, epochs=50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uiKJrAOUc1a"
      },
      "source": [
        "<h3 style='color:purple'>Measure training time on a CPU<h3>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": false,
        "id": "h2ZUnsD6Uc1b",
        "outputId": "ce16e2b9-e0e6-47ce-a86a-303248ed05f1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 2.2248 - accuracy: 0.2036\n",
            "Epoch 2/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.9823 - accuracy: 0.3173\n",
            "Epoch 3/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.9027 - accuracy: 0.3438\n",
            "Epoch 4/50\n",
            "50/50 [==============================] - 2s 42ms/step - loss: 1.8553 - accuracy: 0.3602\n",
            "Epoch 5/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.8222 - accuracy: 0.3708\n",
            "Epoch 6/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.7968 - accuracy: 0.3789\n",
            "Epoch 7/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.7760 - accuracy: 0.3863\n",
            "Epoch 8/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.7583 - accuracy: 0.3926\n",
            "Epoch 9/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.7428 - accuracy: 0.3989\n",
            "Epoch 10/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.7291 - accuracy: 0.4045\n",
            "Epoch 11/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.7165 - accuracy: 0.4076\n",
            "Epoch 12/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.7050 - accuracy: 0.4112\n",
            "Epoch 13/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.6943 - accuracy: 0.4149\n",
            "Epoch 14/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.6844 - accuracy: 0.4180\n",
            "Epoch 15/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.6749 - accuracy: 0.4220\n",
            "Epoch 16/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.6660 - accuracy: 0.4249\n",
            "Epoch 17/50\n",
            "50/50 [==============================] - 2s 42ms/step - loss: 1.6576 - accuracy: 0.4275\n",
            "Epoch 18/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.6495 - accuracy: 0.4303\n",
            "Epoch 19/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.6419 - accuracy: 0.4323\n",
            "Epoch 20/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.6346 - accuracy: 0.4344\n",
            "Epoch 21/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.6276 - accuracy: 0.4366\n",
            "Epoch 22/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.6208 - accuracy: 0.4387\n",
            "Epoch 23/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.6144 - accuracy: 0.4413\n",
            "Epoch 24/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.6083 - accuracy: 0.4435\n",
            "Epoch 25/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.6023 - accuracy: 0.4458\n",
            "Epoch 26/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5966 - accuracy: 0.4473\n",
            "Epoch 27/50\n",
            "50/50 [==============================] - 2s 38ms/step - loss: 1.5911 - accuracy: 0.4492\n",
            "Epoch 28/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5859 - accuracy: 0.4508\n",
            "Epoch 29/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5808 - accuracy: 0.4524\n",
            "Epoch 30/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5759 - accuracy: 0.4545\n",
            "Epoch 31/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5711 - accuracy: 0.4561\n",
            "Epoch 32/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5665 - accuracy: 0.4572\n",
            "Epoch 33/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5621 - accuracy: 0.4586\n",
            "Epoch 34/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5576 - accuracy: 0.4598\n",
            "Epoch 35/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5532 - accuracy: 0.4610\n",
            "Epoch 36/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5490 - accuracy: 0.4628\n",
            "Epoch 37/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5448 - accuracy: 0.4640\n",
            "Epoch 38/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5407 - accuracy: 0.4659\n",
            "Epoch 39/50\n",
            "50/50 [==============================] - 2s 38ms/step - loss: 1.5367 - accuracy: 0.4678\n",
            "Epoch 40/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5328 - accuracy: 0.4690\n",
            "Epoch 41/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5291 - accuracy: 0.4700\n",
            "Epoch 42/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5254 - accuracy: 0.4715\n",
            "Epoch 43/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5217 - accuracy: 0.4726\n",
            "Epoch 44/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5178 - accuracy: 0.4739\n",
            "Epoch 45/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5141 - accuracy: 0.4746\n",
            "Epoch 46/50\n",
            "50/50 [==============================] - 2s 39ms/step - loss: 1.5103 - accuracy: 0.4764\n",
            "Epoch 47/50\n",
            "50/50 [==============================] - 2s 41ms/step - loss: 1.5072 - accuracy: 0.4772\n",
            "Epoch 48/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5039 - accuracy: 0.4780\n",
            "Epoch 49/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.5004 - accuracy: 0.4788\n",
            "Epoch 50/50\n",
            "50/50 [==============================] - 2s 40ms/step - loss: 1.4963 - accuracy: 0.4802\n",
            "1min 57s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)\n"
          ]
        }
      ],
      "source": [
        "%%timeit -n1 -r1\n",
        "with tf.device('/CPU:0'):\n",
        "    cpu_model = get_model()\n",
        "    cpu_model.fit(train_dataset, epochs=50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jS2UReEeUc1b"
      },
      "source": [
        "So comparing DGX A100 GPU performance with its CPU we get following numbers,\n",
        "\n",
        "| Epoch | CPU | GPU  |\n",
        "|:------|:------|:------|\n",
        "| 50 | 1 min 57 sec | 47 sec |\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}