FoodAI_train.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FoodAI_train.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNVM38Lod8YONgkO006ez29",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonathanyin12/Food.AI/blob/master/FoodAI_train.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4123YVhoK3i",
        "colab_type": "text"
      },
      "source": [
        "# **Food.AI Training Notebook**\n",
        "Easy calorie tracking using real-time object detection on Android. \n",
        "\n",
        "**[GitHub](https://github.com/jonathanyin12/Food.AI)**\n",
        "\n",
        "### **Instructions**:\n",
        "* Prior to starting, create a directory in Google Drive called *food_detection*. Add the **[training dataset](https://drive.google.com/file/d/11WC6XPp4kHGN1vEzl_ZRnFla99pxIs33/view)** and **[label_map.pbtxt](https://github.com/jonathanyin12/Food.AI/blob/master/label_map.pbtxt)** to *food_detection*.\n",
        "\n",
        "* Make sure runtime is set to use GPU acceleration\n",
        "\n",
        "* After installing the correct version of NumPy, restart the runtime and run all the cells after.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rCXvr-0MSivk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 372
        },
        "outputId": "7039397d-9154-453c-cac1-2463b7f32458"
      },
      "source": [
        "# Checking gpu\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sun Jun 21 11:57:12 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 450.36.06    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   36C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kvLE33UNCag",
        "colab_type": "text"
      },
      "source": [
        "### **Installing Dependencies**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RHFbvc0TV8Yc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "d16ab71d-d9ac-4596-af8c-9687c15ee894"
      },
      "source": [
        "# IMPORTANT: make sure to restart runtime after running this cell\n",
        "!pip install numpy==1.17.4"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting numpy==1.17.4\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d2/ab/43e678759326f728de861edbef34b8e2ad1b1490505f20e0d1f0716c3bf4/numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl (20.0MB)\n",
            "\u001b[K     |████████████████████████████████| 20.0MB 1.4MB/s \n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: numpy\n",
            "  Found existing installation: numpy 1.18.5\n",
            "    Uninstalling numpy-1.18.5:\n",
            "      Successfully uninstalled numpy-1.18.5\n",
            "Successfully installed numpy-1.17.4\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dIoahiCWSJb0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 407
        },
        "outputId": "ca6f1b8a-391f-445f-9e8a-f89a7f930dbf"
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "\n",
        "!git clone --quiet https://github.com/tensorflow/models.git # download training tools\n",
        "\n",
        "!wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz # download model\n",
        "!tar xvzf ssd_mobilenet_v2_coco_2018_03_29.tar.gz\n",
        " \n",
        "!git clone --quiet https://github.com/zamblauskas/oidv4-toolkit-tfrecord-generator.git # download tfrecord generator tool\n",
        "\n",
        "!apt-get install -qq protobuf-compiler python-tk\n",
        "\n",
        "!pip install -q Cython contextlib2 pillow lxml matplotlib PyDrive pycocotools"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "--2020-06-21 11:58:11--  http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz\n",
            "Resolving download.tensorflow.org (download.tensorflow.org)... 172.217.203.128, 2607:f8b0:400c:c07::80\n",
            "Connecting to download.tensorflow.org (download.tensorflow.org)|172.217.203.128|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 187925923 (179M) [application/x-tar]\n",
            "Saving to: ‘ssd_mobilenet_v2_coco_2018_03_29.tar.gz’\n",
            "\n",
            "ssd_mobilenet_v2_co 100%[===================>] 179.22M   269MB/s    in 0.7s    \n",
            "\n",
            "2020-06-21 11:58:11 (269 MB/s) - ‘ssd_mobilenet_v2_coco_2018_03_29.tar.gz’ saved [187925923/187925923]\n",
            "\n",
            "ssd_mobilenet_v2_coco_2018_03_29/checkpoint\n",
            "ssd_mobilenet_v2_coco_2018_03_29/model.ckpt.meta\n",
            "ssd_mobilenet_v2_coco_2018_03_29/pipeline.config\n",
            "ssd_mobilenet_v2_coco_2018_03_29/saved_model/saved_model.pb\n",
            "ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb\n",
            "ssd_mobilenet_v2_coco_2018_03_29/saved_model/\n",
            "ssd_mobilenet_v2_coco_2018_03_29/saved_model/variables/\n",
            "ssd_mobilenet_v2_coco_2018_03_29/model.ckpt.index\n",
            "ssd_mobilenet_v2_coco_2018_03_29/\n",
            "ssd_mobilenet_v2_coco_2018_03_29/model.ckpt.data-00000-of-00001\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c6d3ccbPSN6G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "outputId": "212485ab-8a57-4c71-cd3f-e243aa763741"
      },
      "source": [
        "%cd /content/models/research\n",
        "!protoc object_detection/protos/*.proto --python_out=.\n",
        "!python setup.py -q build\n",
        "!python setup.py -q install\n",
        "\n",
        "import os\n",
        "os.environ['PYTHONPATH'] += '/content/models/research/:/content/models/research/slim/'\n",
        "\n",
        "%cd /content/models/research/slim\n",
        "!pip install -e .\n",
        "\n",
        "%cd /content/models/research\n",
        "!python object_detection/builders/model_builder_test.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/models/research\n",
            "object_detection/protos/input_reader.proto: warning: Import object_detection/protos/image_resizer.proto but not used.\n",
            "zip_safe flag not set; analyzing archive contents...\n",
            "object_detection.core.__pycache__.preprocessor.cpython-36: module MAY be using inspect.stack\n",
            "object_detection.utils.__pycache__.autoaugment_utils.cpython-36: module MAY be using inspect.stack\n",
            "/content/models/research/slim\n",
            "Obtaining file:///content/models/research/slim\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from slim==0.1) (1.12.0)\n",
            "Collecting tf-slim>=1.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352kB)\n",
            "\u001b[K     |████████████████████████████████| 358kB 2.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from tf-slim>=1.1->slim==0.1) (0.9.0)\n",
            "Installing collected packages: tf-slim, slim\n",
            "  Running setup.py develop for slim\n",
            "Successfully installed slim tf-slim-1.1.0\n",
            "/content/models/research\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VhAQOrINKqy",
        "colab_type": "text"
      },
      "source": [
        "### **Load food dataset and label map from Drive**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EuI0mMxJOb2h",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "eaa622d7-e794-4e87-d392-7875224c76e5"
      },
      "source": [
        "%cd /content/\n",
        "\n",
        "from google.colab import drive\n",
        "from zipfile import ZipFile\n",
        "from shutil import copyfile\n",
        "\n",
        "drive.mount('/content/gdrive')\n",
        "copyfile('/content/gdrive/My Drive/food_detection/label_map.pbtxt', 'label_map.pbtxt')\n",
        "\n",
        "data_path='/content/gdrive/My Drive/food_detection/OIDv4_ToolKit.zip'\n",
        "with ZipFile(data_path, 'r') as zipObj:\n",
        "    zipObj.extractall()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GqQQ7uGQNh4u",
        "colab_type": "text"
      },
      "source": [
        "### **Convert dataset into TFRecords format**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OjS_ckYPOaZV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 641
        },
        "outputId": "dbe943ac-f843-460d-c917-3828c8f3e283"
      },
      "source": [
        "!python oidv4-toolkit-tfrecord-generator/generate-tfrecord.py \\\n",
        "    --classes_file=OIDv4_ToolKit/classes.txt \\\n",
        "    --class_descriptions_file=OIDv4_ToolKit/OID/csv_folder/class-descriptions-boxable.csv \\\n",
        "    --annotations_file=OIDv4_ToolKit/OID/csv_folder/train-annotations-bbox.csv \\\n",
        "    --images_dir=OIDv4_ToolKit/OID/Dataset/train \\\n",
        "    --output_file=train.record\n",
        "\n",
        "!python oidv4-toolkit-tfrecord-generator/generate-tfrecord.py \\\n",
        "    --classes_file=OIDv4_ToolKit/classes.txt \\\n",
        "    --class_descriptions_file=OIDv4_ToolKit/OID/csv_folder/class-descriptions-boxable.csv \\\n",
        "    --annotations_file=OIDv4_ToolKit/OID/csv_folder/validation-annotations-bbox.csv \\\n",
        "    --images_dir=OIDv4_ToolKit/OID/Dataset/validation \\\n",
        "    --output_file=val.record"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:79: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "Classes: {'Bread': 1, 'Pancake': 2, 'Waffle': 3, 'Bagel': 4, 'Muffin': 5, 'Doughnut': 6, 'Hamburger': 7, 'Pizza': 8, 'Sandwich': 9, 'Hot dog': 10, 'French fries': 11, 'Apple': 12, 'Orange': 13, 'Banana': 14, 'Grape': 15}\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:32: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0621 12:01:36.732901 140051544106880 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:32: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "6419 images\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:37: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "W0621 12:01:36.878070 140051544106880 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:37: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:40: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "W0621 12:01:36.909929 140051544106880 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:40: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... done\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:79: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "Classes: {'Bread': 1, 'Pancake': 2, 'Waffle': 3, 'Bagel': 4, 'Muffin': 5, 'Doughnut': 6, 'Hamburger': 7, 'Pizza': 8, 'Sandwich': 9, 'Hot dog': 10, 'French fries': 11, 'Apple': 12, 'Orange': 13, 'Banana': 14, 'Grape': 15}\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:32: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0621 12:02:33.554179 139754588473216 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:32: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "519 images\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:37: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "W0621 12:02:33.596302 139754588473216 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:37: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "WARNING:tensorflow:From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:40: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "W0621 12:02:33.605079 139754588473216 module_wrapper.py:139] From oidv4-toolkit-tfrecord-generator/generate-tfrecord.py:40: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l_rDls2EgQlR",
        "colab_type": "text"
      },
      "source": [
        "### **Edit model configuration files**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wbutwvfdINn-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "\n",
        "filename = '/content/models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config'\n",
        "with open(filename) as f:\n",
        "    s = f.read()\n",
        "with open(filename, 'w') as f:\n",
        "    s = re.sub('90', '15', s) # change number of classes from 90 to 15\n",
        "    s = re.sub('PATH_TO_BE_CONFIGURED/model.ckpt', '/content/ssd_mobilenet_v2_coco_2018_03_29/model.ckpt', s) # pass location of model checkpoint\n",
        "    s = re.sub('PATH_TO_BE_CONFIGURED/mscoco_train.record-\\?\\?\\?\\?\\?-of-00100', '/content/train.record', s) # pass location of train tfrecod\n",
        "    s = re.sub('PATH_TO_BE_CONFIGURED/mscoco_val.record-\\?\\?\\?\\?\\?-of-00010', '/content/val.record', s) # pass location of validation tfrecord\n",
        "    s = re.sub('PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt', '/content/label_map.pbtxt', s) # pass location of label map\n",
        "\n",
        "    f.write(s)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6-wLScivf5-x",
        "colab_type": "text"
      },
      "source": [
        "### **Train model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "cellView": "both",
        "id": "T_XfV1HTQp_f",
        "colab": {}
      },
      "source": [
        "!python models/research/object_detection/model_main.py \\\n",
        "    --pipeline_config_path=/content/models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config \\\n",
        "    --model_dir=/content/gdrive/My\\ Drive/food_detection/model_checkpoints"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tD0SbAMjmjsT",
        "colab_type": "text"
      },
      "source": [
        "### **Export model to frozen TensorFlow Lite graph**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DIM3xJw2HkyO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lst = os.listdir('/content/gdrive/My Drive/food_detection/model_checkpoints')\n",
        "lf = filter(lambda k: 'model.ckpt-' in k, lst)\n",
        "last_model = sorted(lf)[-1].replace('.meta', '')\n",
        "\n",
        "!python /content/models/research/object_detection/export_tflite_ssd_graph.py \\\n",
        "    --pipeline_config_path=/content/models/research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config \\\n",
        "    --trained_checkpoint_prefix=/content/gdrive/My\\ Drive/food_detection/model_checkpoints/$last_model \\\n",
        "    --output_directory=/content/gdrive/My\\ Drive/food_detection/model_checkpoints/tflite_model \\\n",
        "    --add_postprocessing_op=true"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6m_Mxhbgm02f",
        "colab_type": "text"
      },
      "source": [
        "### **Convert the frozen graph to a TFLite model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tOPsn-TjQvZv",
        "colab": {}
      },
      "source": [
        "%cd /content/gdrive/My\\ Drive/food_detection/model_checkpoints/tflite_model\n",
        "\n",
        "!tflite_convert \\\n",
        "    --output_file=food_detect.tflite \\\n",
        "    --graph_def_file=tflite_graph.pb \\\n",
        "    --input_arrays=normalized_input_image_tensor \\\n",
        "    --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3'\\\n",
        "    --input_shape=1,300,300,3 \\\n",
        "    --allow_custom_ops"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}