triplet_network_and_one_shot_learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "triplet_network_and_one_shot_learning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "p-gkaM1tCThc"
      },
      "source": [
        "# Triplet networks and one-shot learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZqdQgnI5AuN5"
      },
      "source": [
        "## Import packages and mount data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8QdDDUEIAuN6",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import backend as K\n",
        "from tensorflow.keras.layers import Input, Conv2D, Lambda, Dense, Flatten, MaxPooling2D, Dropout, Concatenate, BatchNormalization, concatenate, ReLU, LeakyReLU\n",
        "from tensorflow.keras.models import Model, Sequential\n",
        "from tensorflow.keras.regularizers import l2\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.losses import binary_crossentropy\n",
        "import numpy as np\n",
        "import os\n",
        "import pickle\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.utils import shuffle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YzhGJvL2A7aL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e63baee3-1074-4308-c3fc-e7893f66abce"
      },
      "source": [
        "# mount the data needed to drive folder so we can use them in colab, data is stored in google drive\n",
        "from google.colab import drive\n",
        "!mkdir drive\n",
        "drive.mount('drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mkdir: cannot create directory ‘drive’: File exists\n",
            "Drive already mounted at drive; to attempt to forcibly remount, call drive.mount(\"drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NEy5u5WBAuN_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        },
        "outputId": "290fb7c1-e9c8-4daf-fae7-d053d76a6616"
      },
      "source": [
        "PATH = os.path.join(\"drive\",\"My Drive\",\"omniglot\")\n",
        "\n",
        "with open(os.path.join(PATH, \"omniglot_train.p\"), \"rb\") as f:\n",
        "    (X_train, c_train) = pickle.load(f)\n",
        "\n",
        "with open(os.path.join(PATH, \"omniglot_test.p\"), \"rb\") as f:\n",
        "    (X_test, c_test) = pickle.load(f)\n",
        "\n",
        "print(\"X_train shape:\", X_train.shape)\n",
        "print(\"X_test shape:\", X_test.shape)\n",
        "print(\"\")\n",
        "print(\"training alphabets\")\n",
        "print([key for key in c_train.keys()])\n",
        "print(\"test alphabets:\")\n",
        "print([key for key in c_test.keys()])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_train shape: (964, 20, 105, 105)\n",
            "X_test shape: (659, 20, 105, 105)\n",
            "\n",
            "training alphabets\n",
            "['Braille', 'Anglo-Saxon_Futhorc', 'Tifinagh', 'Grantha', 'Burmese_(Myanmar)', 'Mkhedruli_(Georgian)', 'Latin', 'Ojibwe_(Canadian_Aboriginal_Syllabics)', 'Balinese', 'Malay_(Jawi_-_Arabic)', 'Early_Aramaic', 'Korean', 'Japanese_(hiragana)', 'Armenian', 'Cyrillic', 'Hebrew', 'Syriac_(Estrangelo)', 'Japanese_(katakana)', 'Blackfoot_(Canadian_Aboriginal_Syllabics)', 'N_Ko', 'Alphabet_of_the_Magi', 'Inuktitut_(Canadian_Aboriginal_Syllabics)', 'Greek', 'Bengali', 'Tagalog', 'Futurama', 'Arcadian', 'Gujarati', 'Asomtavruli_(Georgian)', 'Sanskrit']\n",
            "test alphabets:\n",
            "['ULOG', 'Atemayar_Qelisayer', 'Ge_ez', 'Gurmukhi', 'Tengwar', 'Keble', 'Malayalam', 'Oriya', 'Kannada', 'Mongolian', 'Angelic', 'Atlantean', 'Syriac_(Serto)', 'Aurek-Besh', 'Avesta', 'Glagolitic', 'Sylheti', 'Tibetan', 'Manipuri', 'Old_Church_Slavonic_(Cyrillic)']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QP79HYQrXD2k"
      },
      "source": [
        "## Building the triplet network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1WHpL8iHAuOH"
      },
      "source": [
        "We will define a triplet Network for use with the Omniglot dataset. Each branch of the triplet  is a \"convnet\" model that transforms data to an embeddings space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "GQNaMa8hXD2l",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 901
        },
        "outputId": "902634f3-bda3-42da-8428-a760ea0c77ea"
      },
      "source": [
        "# define a convnet model to transforms data to an embeddings space. \n",
        "input_shape = (105, 105, 1)\n",
        "\n",
        "# The architecture is similar to that in the paper (Koch et al., \"Siamese Neural Networks for One-shot Image Recognition\"), \n",
        "# but we include dropout and batch normalization to improve generalization and speed up training.\n",
        "convnet = Sequential()\n",
        "convnet.add(Conv2D(64, (3,3), input_shape=input_shape))\n",
        "convnet.add(BatchNormalization())\n",
        "convnet.add(ReLU())\n",
        "convnet.add(MaxPooling2D((2,2)))\n",
        "convnet.add(Dropout(0.2))\n",
        "\n",
        "convnet.add(Conv2D(128, (3,3)))\n",
        "convnet.add(BatchNormalization())\n",
        "convnet.add(ReLU())\n",
        "convnet.add(MaxPooling2D((2,2)))\n",
        "convnet.add(Dropout(0.2))\n",
        "\n",
        "convnet.add(Conv2D(128, (3,3)))\n",
        "convnet.add(BatchNormalization())\n",
        "convnet.add(ReLU())\n",
        "convnet.add(MaxPooling2D((2,2)))\n",
        "convnet.add(Dropout(0.2))\n",
        "\n",
        "convnet.add(Conv2D(256, (3,3)))\n",
        "convnet.add(BatchNormalization())\n",
        "convnet.add(ReLU())\n",
        "convnet.add(MaxPooling2D((2,2)))\n",
        "convnet.add(Dropout(0.2))\n",
        "\n",
        "convnet.add(Flatten())\n",
        "\n",
        "convnet.add(Dense(1024, activation=\"linear\"))\n",
        "\n",
        "convnet._name = \"leg\"\n",
        "\n",
        "convnet.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"leg\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_20 (Conv2D)           (None, 103, 103, 64)      640       \n",
            "_________________________________________________________________\n",
            "batch_normalization_20 (Batc (None, 103, 103, 64)      256       \n",
            "_________________________________________________________________\n",
            "re_lu_20 (ReLU)              (None, 103, 103, 64)      0         \n",
            "_________________________________________________________________\n",
            "max_pooling2d_20 (MaxPooling (None, 51, 51, 64)        0         \n",
            "_________________________________________________________________\n",
            "dropout_20 (Dropout)         (None, 51, 51, 64)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_21 (Conv2D)           (None, 49, 49, 128)       73856     \n",
            "_________________________________________________________________\n",
            "batch_normalization_21 (Batc (None, 49, 49, 128)       512       \n",
            "_________________________________________________________________\n",
            "re_lu_21 (ReLU)              (None, 49, 49, 128)       0         \n",
            "_________________________________________________________________\n",
            "max_pooling2d_21 (MaxPooling (None, 24, 24, 128)       0         \n",
            "_________________________________________________________________\n",
            "dropout_21 (Dropout)         (None, 24, 24, 128)       0         \n",
            "_________________________________________________________________\n",
            "conv2d_22 (Conv2D)           (None, 22, 22, 128)       147584    \n",
            "_________________________________________________________________\n",
            "batch_normalization_22 (Batc (None, 22, 22, 128)       512       \n",
            "_________________________________________________________________\n",
            "re_lu_22 (ReLU)              (None, 22, 22, 128)       0         \n",
            "_________________________________________________________________\n",
            "max_pooling2d_22 (MaxPooling (None, 11, 11, 128)       0         \n",
            "_________________________________________________________________\n",
            "dropout_22 (Dropout)         (None, 11, 11, 128)       0         \n",
            "_________________________________________________________________\n",
            "conv2d_23 (Conv2D)           (None, 9, 9, 256)         295168    \n",
            "_________________________________________________________________\n",
            "batch_normalization_23 (Batc (None, 9, 9, 256)         1024      \n",
            "_________________________________________________________________\n",
            "re_lu_23 (ReLU)              (None, 9, 9, 256)         0         \n",
            "_________________________________________________________________\n",
            "max_pooling2d_23 (MaxPooling (None, 4, 4, 256)         0         \n",
            "_________________________________________________________________\n",
            "dropout_23 (Dropout)         (None, 4, 4, 256)         0         \n",
            "_________________________________________________________________\n",
            "flatten_5 (Flatten)          (None, 4096)              0         \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 1024)              4195328   \n",
            "=================================================================\n",
            "Total params: 4,714,880\n",
            "Trainable params: 4,713,728\n",
            "Non-trainable params: 1,152\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RXVuWeCsAuOI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 425
        },
        "outputId": "a8e3bdf2-8af6-4ffe-d8b2-7323f47ffb40"
      },
      "source": [
        "# The anchor, positive, negative image are merged together, as the input of the triplet network, then got split to get each one's neural codes.\n",
        "generated = Input(shape=(3, 105, 105, 1), name='input')\n",
        "\n",
        "anchor = Lambda(lambda x: x[:, 0])(generated)\n",
        "pos = Lambda(lambda x: x[:, 1])(generated)\n",
        "neg = Lambda(lambda x: x[:, 2])(generated)\n",
        "\n",
        "# merge the anchor, positive, negative embedding together, \n",
        "# let the merged layer be the output of triplet network\n",
        "anchor_embedding = convnet(anchor)\n",
        "pos_embedding = convnet(pos)\n",
        "neg_embedding = convnet(neg)  \n",
        "\n",
        "merged_output = concatenate([anchor_embedding, pos_embedding, neg_embedding], axis=-1, name='merged_layer')\n",
        "\n",
        "triplet_net = Model(inputs=generated, outputs=merged_output)\n",
        "triplet_net.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_142\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input (InputLayer)              [(None, 3, 105, 105, 0                                            \n",
            "__________________________________________________________________________________________________\n",
            "lambda_12 (Lambda)              (None, 105, 105, 1)  0           input[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "lambda_13 (Lambda)              (None, 105, 105, 1)  0           input[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "lambda_14 (Lambda)              (None, 105, 105, 1)  0           input[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "leg (Sequential)                (None, 1024)         4714880     lambda_12[0][0]                  \n",
            "                                                                 lambda_13[0][0]                  \n",
            "                                                                 lambda_14[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "merged_layer (Concatenate)      (None, 3072)         0           leg[1][0]                        \n",
            "                                                                 leg[2][0]                        \n",
            "                                                                 leg[3][0]                        \n",
            "==================================================================================================\n",
            "Total params: 4,714,880\n",
            "Trainable params: 4,713,728\n",
            "Non-trainable params: 1,152\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A-XyrIANAuOM"
      },
      "source": [
        "## Defining the triplet loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "mZ3v2Z0RAuON",
        "colab": {}
      },
      "source": [
        "# Notice that the ground truth variable is not used for loss calculation. \n",
        "# It is used as a function argument to by-pass some Keras functionality.\n",
        "# This is because the network structure already implies the ground truth for the anchor image with the \"positive\" image.\n",
        "import tensorflow as tf\n",
        "def triplet_loss(ground_truth, network_output):\n",
        "\n",
        "    anchor, positive, negative = tf.split(network_output, num_or_size_splits=3, axis=1)        \n",
        "\n",
        "    # This is an easy implementation, but also a very inefficient one because it uses offline triplet mining (https://omoindrot.github.io/triplet-loss)\n",
        "    positive_distance = tf.reduce_sum(tf.square(anchor - positive), 1)\n",
        "    negative_distance = tf.reduce_sum(tf.square(anchor - negative), 1)\n",
        "\n",
        "    margin = 2000\n",
        "    loss = tf.maximum(positive_distance - negative_distance + margin, 0.0)\n",
        "    loss = tf.reduce_mean(loss)\n",
        " \n",
        "    return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "H7Wo8uzTXD2v"
      },
      "source": [
        "## Selecting triplets for training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tSSr9IzTAuOX"
      },
      "source": [
        "#### Different  selection method\n",
        "\n",
        "We have two different options for the triplet selection method, and we will compare the model performance under these two methods after building our model.\n",
        "\n",
        "(1) Random  triplets selection, including the following steps:\n",
        "* Pick one random class for anchor\n",
        "* Pick two different random picture for this class, as the anchor and positive images\n",
        "* Pick another class for Negative, different from anchor_class\n",
        "* Pick one random picture from the negative class.\n",
        "\n",
        "(2) Hard triplets selection. For easy implement, for a picked anchor, positive pair, we will choose the hardest negative to form a hard triplet, that means, after picking an anchor, positive image, we will choose the negative image which is nearest from anchor image from a negative class, ie: \"- d(a,n)\"  can get the maximum value. The whole process including the following steps:\n",
        "* Pick one random class for anchor\n",
        "* Pick two different random picture for this class, as an anchor and positive images\n",
        "* Pick another class for negative, different from anchor_class\n",
        "* Pick one hardest picture from the negative class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HcxMsmACAuOY",
        "colab": {}
      },
      "source": [
        "# Notice that the returned  1 * np.zeros(batch_size) is to by-pass some Keras functionality, corresponding to ground_truth in tripletloss\n",
        "# We use a variable hard_selection to control which method we are going to use. If we set hard_selection == False, we will select triplets random,If we set the variable hard_selection == True, we will select hard triplets.\n",
        "def get_batch(batch_size, X, hard_selection):\n",
        "    # Create a subset of the model that basically represents a \"leg\" of the model\n",
        "    subset_model = Model(inputs=triplet_net.get_layer(\"leg\").get_input_at(0), \n",
        "                         outputs=triplet_net.get_layer(\"leg\").get_output_at(0))\n",
        "\n",
        "    while True:\n",
        "        n_classes, n_examples, w, h = X.shape\n",
        "        \n",
        "        # initialize result\n",
        "        triplets = []\n",
        "\n",
        "        for i in range(batch_size):\n",
        "            triplet = [[], [], []]\n",
        "\n",
        "            # Pick one random class for anchor\n",
        "            anchor_class = np.random.randint(0, n_classes)\n",
        "\n",
        "            # Pick two different random pics for this class => idx_A and idx_P\n",
        "            [idx_A, idx_P] = np.random.choice(n_examples, size=2, replace=False)\n",
        "            #print(f\"Anchor class: {anchor_class}, idx_A: {idx_A}, idx_P: {idx_P}\")\n",
        "            \n",
        "            # Pick another class for negative, different from anchor_class\n",
        "            negative_class = np.random.choice(np.setdiff1d(range(0, n_classes), anchor_class))\n",
        "            # print(f\"Negative class: {negative_class}, shape: {X[negative_class].shape}\")\n",
        "\n",
        "            if not hard_selection:\n",
        "                # Pick a random pic from this negative class => N \n",
        "                idx_N = np.random.choice(n_examples, size=1, replace=False)\n",
        "\n",
        "            else:\n",
        "                # Pick a hardest pic from this negative class => N\n",
        "                \n",
        "                # Get the embedding of the anchor image\n",
        "                anchor_img = subset_model.predict(np.expand_dims(X[anchor_class][idx_A], axis=0))\n",
        "\n",
        "                # Make a prediction for all images in the negative class\n",
        "                neg_imgs = subset_model.predict(np.expand_dims(X[negative_class], axis=0).reshape(20, 105, 105, 1))\n",
        "                \n",
        "                # Compute the distance (note that we use the l2 distance) between the anchor and negative img embeddings\n",
        "                distances = [np.linalg.norm(anchor_img - neg_img) for neg_img in neg_imgs]\n",
        "\n",
        "                # Pick the image with the nearest distance as the \"hard\" image\n",
        "                idx_N = np.argsort(distances)[0]\n",
        "\n",
        "            triplet[0] = X[anchor_class][idx_A].reshape(w, h, 1)\n",
        "            triplet[1] = X[anchor_class][idx_P].reshape(w, h, 1)\n",
        "            triplet[2]=  X[negative_class][idx_N].reshape(w, h, 1)\n",
        "            triplets.append(triplet)\n",
        "\n",
        "        yield np.array(triplets), 1 * np.zeros(batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LP1oojLhXD2z"
      },
      "source": [
        "## One-shot learning with different selection method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RBgoMDwMAuOh",
        "colab": {}
      },
      "source": [
        "def make_oneshot_task(N, X, c, language=None):\n",
        "    \"\"\"Create pairs of (test image, support set image) with ground truth, for testing N-way one-shot learning.\"\"\"\n",
        "    n_classes, n_examples, w, h = X.shape\n",
        "    indices = np.random.randint(0, n_examples, size=(N,))\n",
        "    if language is not None:\n",
        "        low, high = c[language]\n",
        "        if N > high - low:\n",
        "            raise ValueError(\"This language ({}) has less than {} letters\".format(language, N))\n",
        "        categories = np.random.choice(range(low,high), size=(N,), replace=False)\n",
        "    else:  # if no language specified just pick a bunch of random letters\n",
        "        categories = np.random.choice(range(n_classes), size=(N,), replace=False)            \n",
        "    true_category = categories[0]\n",
        "    ex1, ex2 = np.random.choice(n_examples, replace=False, size=(2,))\n",
        "    test_image = np.asarray([X[true_category, ex1, :, :]]*N).reshape(N, w, h, 1)\n",
        "    support_set = X[categories, indices, :, :]\n",
        "    support_set[0, :, :] = X[true_category, ex2]\n",
        "    support_set = support_set.reshape(N, w, h, 1)\n",
        "    targets = np.zeros((N,))\n",
        "    targets[0] = 1\n",
        "    targets, test_image, support_set = shuffle(targets, test_image, support_set)\n",
        "    pairs = [test_image, support_set]\n",
        "    return pairs, targets"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rk5F3ffuAuOl",
        "colab": {}
      },
      "source": [
        "def test_oneshot(model, X, c, N=20, k=250, language=None, verbose=True):     \n",
        "    \"\"\"Test average N-way oneshot learning accuracy of a siamese neural net over k one-shot tasks.\"\"\"\n",
        "    n_correct = 0\n",
        "    \n",
        "    if verbose:\n",
        "        print(\"Evaluating model on {} random {}-way one-shot learning tasks ...\".format(k, N))\n",
        "\n",
        "    for i in range(k):\n",
        "        # Create a one-shot task \n",
        "        inputs, targets = make_oneshot_task(N, X, c, language=language)\n",
        "\n",
        "        # 1. For a given one-shot task, obtain embeddings for the test image as well as the support set. \n",
        "        test_img = model.predict(inputs[0])\n",
        "        support_set = model.predict(inputs[1])\n",
        "        # Note that we use the l2 distance to compute the distances\n",
        "        distances = [np.linalg.norm(x-y) for x,y in zip(test_img, support_set)]\n",
        "        \n",
        "        # 2. Pick the image from the support set that is closest (in L2-distance) to the test image as your one-shot prediction.\n",
        "        if np.argmin(distances) == np.argmax(targets):\n",
        "            n_correct += 1\n",
        "\n",
        "    percent_correct = (100.0 * n_correct / k)\n",
        "    \n",
        "    if verbose:\n",
        "        print(\"Got an average of {}% accuracy for {}-way one-shot learning\".format(percent_correct, N))\n",
        "    return percent_correct"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "snKKGhIoXD27"
      },
      "source": [
        "## Evaluate one-shot learning with  random triplets selection"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tsn0z4DsxT48",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train(model, X_train, hard_selection=False, batch_size=64, steps_per_epoch=100, epochs=1):\n",
        "    model.fit(get_batch(batch_size, X_train, hard_selection), steps_per_epoch=steps_per_epoch, epochs=epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YLlvr6TRXD2-"
      },
      "source": [
        "## Evaluate one-shot learning with  hard triplets selection"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MV-1lr34Brg0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7a07d783-bb92-45fc-d3f7-519c39f34349"
      },
      "source": [
        "# Random triplet selection\n",
        "triplet_net.compile(loss=triplet_loss, optimizer=Adam(lr=0.0001))\n",
        "loops = 20\n",
        "best_acc_random = 0\n",
        "for i in range(loops):\n",
        "    print(\"=== Training loop {} ===\".format(i+1))\n",
        "    # === ADD CODE HERE ===\n",
        "    train(triplet_net, X_train, hard_selection=False, batch_size=64, steps_per_epoch=100, epochs=1)\n",
        "    subset_model = Model(inputs=triplet_net.get_layer(\"leg\").get_input_at(0), \n",
        "                         outputs=triplet_net.get_layer(\"leg\").get_output_at(0))\n",
        "    test_acc = test_oneshot(subset_model, X_test, c_test)\n",
        "\n",
        "    if test_acc >= best_acc_random:\n",
        "        print(\"********* New best one-shot accuracy, saving model ********\")\n",
        "        triplet_net.save(os.path.join(\".\", \"triplet_net_with_random_selection.h5\"))\n",
        "        best_acc_random = test_acc"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "=== Training loop 1 ===\n",
            "100/100 [==============================] - 33s 329ms/step - loss: 792.4438\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 44.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 2 ===\n",
            "100/100 [==============================] - 33s 328ms/step - loss: 530.6794\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 55.2% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 3 ===\n",
            "100/100 [==============================] - 33s 325ms/step - loss: 462.4393\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 54.4% accuracy for 20-way one-shot learning\n",
            "=== Training loop 4 ===\n",
            "100/100 [==============================] - 33s 326ms/step - loss: 400.8351\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 60.0% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 5 ===\n",
            "100/100 [==============================] - 33s 328ms/step - loss: 291.1412\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 54.4% accuracy for 20-way one-shot learning\n",
            "=== Training loop 6 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 254.4511\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 63.2% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 7 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 238.0981\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 57.2% accuracy for 20-way one-shot learning\n",
            "=== Training loop 8 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 222.7442\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 67.6% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 9 ===\n",
            "100/100 [==============================] - 33s 331ms/step - loss: 217.6014\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 62.8% accuracy for 20-way one-shot learning\n",
            "=== Training loop 10 ===\n",
            "100/100 [==============================] - 33s 326ms/step - loss: 211.1373\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 69.6% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 11 ===\n",
            "100/100 [==============================] - 33s 329ms/step - loss: 190.2452\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 63.2% accuracy for 20-way one-shot learning\n",
            "=== Training loop 12 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 195.6187\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 70.8% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 13 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 163.9812\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 69.2% accuracy for 20-way one-shot learning\n",
            "=== Training loop 14 ===\n",
            "100/100 [==============================] - 33s 327ms/step - loss: 171.1250\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 67.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 15 ===\n",
            "100/100 [==============================] - 33s 328ms/step - loss: 176.3309\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 71.2% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 16 ===\n",
            "100/100 [==============================] - 33s 328ms/step - loss: 158.0013\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 73.6% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 17 ===\n",
            "100/100 [==============================] - 33s 329ms/step - loss: 143.0078\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 70.8% accuracy for 20-way one-shot learning\n",
            "=== Training loop 18 ===\n",
            "100/100 [==============================] - 33s 326ms/step - loss: 144.1058\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 68.0% accuracy for 20-way one-shot learning\n",
            "=== Training loop 19 ===\n",
            "100/100 [==============================] - 32s 324ms/step - loss: 148.3192\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 78.0% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 20 ===\n",
            "100/100 [==============================] - 33s 326ms/step - loss: 147.8568\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 78.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BACJKY8kTeF7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6ee3e7e0-fa59-4384-96c5-769955766545"
      },
      "source": [
        "# Hard triplet selection\n",
        "triplet_net.compile(loss=triplet_loss, optimizer=Adam(lr=0.0001))\n",
        "loops = 20\n",
        "best_acc_hard = 0\n",
        "for i in range(loops):\n",
        "    print(\"=== Training loop {} ===\".format(i+1))\n",
        "    # === ADD CODE HERE ===\n",
        "    train(triplet_net, X_train, hard_selection=True, batch_size=64, steps_per_epoch=100, epochs=1)\n",
        "    subset_model = Model(inputs=triplet_net.get_layer(\"leg\").get_input_at(0), \n",
        "                         outputs=triplet_net.get_layer(\"leg\").get_output_at(0))\n",
        "    test_acc = test_oneshot(subset_model, X_test, c_test)\n",
        "\n",
        "    if test_acc >= best_acc_hard:\n",
        "        print(\"********* New best one-shot accuracy, saving model ********\")\n",
        "        triplet_net.save(os.path.join(\".\", \"triplet_net_with_hard_selection.h5\"))\n",
        "        best_acc_hard = test_acc"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "=== Training loop 1 ===\n",
            "100/100 [==============================] - 311s 3s/step - loss: 1424.7808\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 47.6% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 2 ===\n",
            "100/100 [==============================] - 319s 3s/step - loss: 1084.0181\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 60.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 3 ===\n",
            "100/100 [==============================] - 325s 3s/step - loss: 771.7866\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 62.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 4 ===\n",
            "100/100 [==============================] - 320s 3s/step - loss: 637.5370\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 70.0% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 5 ===\n",
            "100/100 [==============================] - 313s 3s/step - loss: 555.5643\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 68.8% accuracy for 20-way one-shot learning\n",
            "=== Training loop 6 ===\n",
            "100/100 [==============================] - 314s 3s/step - loss: 547.0103\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 69.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 7 ===\n",
            "100/100 [==============================] - 312s 3s/step - loss: 510.5052\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 71.6% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 8 ===\n",
            "100/100 [==============================] - 330s 3s/step - loss: 454.9717\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 72.8% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 9 ===\n",
            "100/100 [==============================] - 322s 3s/step - loss: 463.9086\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 72.4% accuracy for 20-way one-shot learning\n",
            "=== Training loop 10 ===\n",
            "100/100 [==============================] - 312s 3s/step - loss: 412.5074\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 74.0% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 11 ===\n",
            "100/100 [==============================] - 309s 3s/step - loss: 390.2656\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 73.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 12 ===\n",
            "100/100 [==============================] - 312s 3s/step - loss: 353.6987\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 80.0% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 13 ===\n",
            "100/100 [==============================] - 316s 3s/step - loss: 339.3546\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 78.0% accuracy for 20-way one-shot learning\n",
            "=== Training loop 14 ===\n",
            "100/100 [==============================] - 318s 3s/step - loss: 325.2707\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 82.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n",
            "=== Training loop 15 ===\n",
            "100/100 [==============================] - 317s 3s/step - loss: 326.9020\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 79.2% accuracy for 20-way one-shot learning\n",
            "=== Training loop 16 ===\n",
            "100/100 [==============================] - 313s 3s/step - loss: 301.4644\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 79.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 17 ===\n",
            "100/100 [==============================] - 311s 3s/step - loss: 277.7395\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 81.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 18 ===\n",
            "100/100 [==============================] - 313s 3s/step - loss: 268.7421\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 79.6% accuracy for 20-way one-shot learning\n",
            "=== Training loop 19 ===\n",
            "100/100 [==============================] - 322s 3s/step - loss: 263.1913\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 78.8% accuracy for 20-way one-shot learning\n",
            "=== Training loop 20 ===\n",
            "100/100 [==============================] - 316s 3s/step - loss: 239.4143\n",
            "Evaluating model on 250 random 20-way one-shot learning tasks ...\n",
            "Got an average of 84.4% accuracy for 20-way one-shot learning\n",
            "********* New best one-shot accuracy, saving model ********\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3bBzgWR5zltZ",
        "colab_type": "text"
      },
      "source": [
        "## Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IElMzWPd3nK8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "56152d08-a9b3-4a64-cc47-87f03466d77a"
      },
      "source": [
        "print(f\"Best accuracy using random triplets: {best_acc_random}\\nBest accuracy using hard triplets: {best_acc_hard}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Best accuracy using random triplets: 78.4\n",
            "Best accuracy using hard triplets: 84.4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5JnVHH4xCn_6",
        "colab_type": "text"
      },
      "source": [
        "Before we evaluate our model and compare the performance of random triplets against hard triplets, we will first provide some motivation regarding our model architecture (specifically, the architecture of a single \"leg\" of the model) and the margin we used for comparing the random triplets against the hard triplets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8jg3FCpOClVr",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "### Determining the model\n",
        "Through experimenting, we observed that the architecture of one single \"leg\" of the model has a significant impact on the resulting accuracy. Initially, we started with simple baseline model. We continued by extending this model and varying the number of filters in each convolutional layer. We tried varying the number of convolutional blocks, including batch normalization, max pooling, and dropout. We use max-pooling layers to reduce overfitting and to keep the number of parameters on the low side, which in turn decreases the training time (we explain later why this is important). We use dropout to reduce overfitting since it is a straightforward and effective way to do this. We have also tried experimenting with the optimizer and the batch sizes and stick with Adam with a batch size of 64 since this combination seemed to perform well in most cases. As our final layer, we use a linear activation function since we want to preserve the absolute distances between embeddings. Furthermore, we noticed that other activation functions (e.g., sigmoid) did not significantly speed up training.\n",
        "\n",
        "While we designed our model architecture (the \"legs\"), we wanted to keep the model relatively simple; we argue that this assignment was not given to us to spend hours and hours on finding the best model for the \"legs\", but rather to get acquainted with the concepts and tinker with aspects such as the margin, the model, and the selection method to get insights on how these networks work. Furthermore, keeping our model simple ensured that we \"only\" a couple of million parameters. This was convenient since it resulted in shorter training times, which was especially useful for hard triplet selection.\n",
        "\n",
        "Ultimately, we found that our current model performed the \"best\". It is hard to argue what the \"best\" is in this environment since several processes are performed at \"random\" (e.g., picking the anchor and positive image class, picking the anchor and positive images, and picking the negative image class). For that reason, we tried to use the hard triplet selection method to verify whether a model performed better or not. We limited the number of loops to three and used this to decide what model performed best. Since we did not want to put all our trust in three loops, whose result can still be somewhat \"random\" because the negative image class is picked at \"random\", we also decided to run the model with the random triplets selection method for 10 loops. We are aware that this method is not perfect for comparing models. However, we observed that some models resulted in accuracies between 30% and 40% in most of the loops. In contrast, other models resulted in accuracies that were consistently at least 60% or higher after several loops."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TT7JDWLUpoQb",
        "colab_type": "text"
      },
      "source": [
        "### Determining the margin\n",
        "It is interesting to tinker with the margin of our triplet loss and see how varying margins influence the results. From the lecture notes, we know that if the margin is low, we have relatively few triplets $(x_a, x_p, x_n)$ that are semi-hard, that is, $||f(x_p) - f(x_a)||^2 < ||f(x_n) - f(x_a)||^2 < ||f(x_p) - f(x_a)||^2 + \\alpha$, compared to when the margin is higher. Such a reduction in the number of semi-hard triplets implies an increase in the number of easy negatives. From the lecture notes, we know that easy negatives are less informative and will not contribute any gradients or fewer gradients to our training. This means that the training is less effective, and our model is expected to not achieve as good results as it could.\n",
        "\n",
        "On the other hand, what happens if the margin is \"too large\"? To make life easier, assume that the optimal value of the margin (the margin value that leads to the selection of the best triplets) equals 1500. If we now pick a margin that equals 2000, we allow more triplets to be semi-hard negatives than we would have if we had chosen the optimal margin of 1500. This means that we allow triplets that are easy negatives when we consider the optimal margin to be semi-hard negatives with our margin. Since we now use these new semi-hard triplets for training our model, we basically use triplets that are easy negatives in our optimal situation. By definition of our optimal margin, these new semi-hard triplets that are introduced by setting the margin to 2000 carry less information than the semi-hard triplets that exist when using the optimal margin. Hence, we expect our model to learn less from these new semi-hard triplets introduced by increasing the margin higher than the optimal margin. As a result, we expect our model to learn less and, on average, the accuracies to be lower. The more we increase our margin (i.e., the larger the difference between the optimal and our margin), the more \"less informative\" semi-hard triplets we use for training. Hence, we increase the margin more and more, and so we expect the performance of the model to go down as we use more and more \"less informative\" semi-hard triplets.\n",
        "\n",
        "Therefore, we expect that there is a sweet spot for the optimal margin value. We played with small, medium, and large margin values and compared the obtained results. We determined what small, medium, and large margins are in our case by comparing the distances between the anchor and positive, and anchor and negative embeddings using the `tf.print()` statement. We have attempted quite some margin values, and we, indeed, observed that for small margin values such as 0, 50, and 100, the accuracy was not as high as the maximum accuracies we observed, which were obtained with a margin of 2.000. On the other side of the spectrum, we observed that for large margin values such as 5.000 and 10.000, the accuracy was also not as high as the maximum accuracies we observed for a margin of 2.000. In the end, the best margin value we found was 2.000, which resulted in an of 78.4% using the random triplet selection method and an accuracy of 84.4% using the hard triplet selection method. Note that these accuracies are the highest accuracies observed in 20 loops. We also observed that for the random triplet selection method, many of the accuracies obtained after 12 loops were higher than 70%. On the other hand, after 12 loops for the hard triplet selection method, we observed that most accuracies were higher than 79%.\n",
        "\n",
        "It must also be noted that determining the margin value is difficult since the embedding changes as the training proceeds. In [3], Sun et al. also mention that determining a proper margin was one of their practical difficulties. Is there a smarter way to pick the margin? It seems like there is. Zhang et al. [2] propose a novel multi-stage training strategy that learns incremental triplet margin and improves triplet loss effectively. It would be interesting to explore this concept in the future.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuqEt2ypCwU7",
        "colab_type": "text"
      },
      "source": [
        "### Comparing random triplets to hard triplets.\n",
        "To reiterate, under the random triplet selection method, we achieved an accuracy of 78.4%. Under the hard triplet selection method, we achieved an accuracy of 84.4%. Comparing the highest accuracies obtained is not necessarily the best comparison since it is possible the random triplet selection results in an extremely high accuracy simply by luck (since picking the negative image class and the negative image in that class is done at random). Overall, however, we indeed observe that hard triplet selection achieves higher accuracies on a consistent basis (i.e., it does not perform better in 1 of the 20 loops, but it performs better in most loops), which makes sense since the model is given triplets that are, in most cases, more informative, theoretically speaking.\n",
        "\n",
        "However, it must also be noted that the hard triplet selection required significantly longer to train, namely around 310 seconds. This is significantly longer than the 33 seconds for random triplet selection. This would be a serious disadvantage if we were to train our model for longer periods.\n",
        "\n",
        "In the future, it would be interesting to explore the strategy proposed by Zhang et al. [2] to pick a margin. Furthermore, it would be exciting to implement online triplet mining and see how that compares to our setup.\n",
        "\n",
        "[2] https://arxiv.org/pdf/1812.06576.pdf\n",
        "\n",
        "[3] https://arxiv.org/pdf/1406.4773.pdf"
      ]
    }
  ]
}