visual-image-search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual image search\n",
    "_**Using a Convolutional Neural Net and Elasticsearch k-Nearest Neighbors Index to retrieve visually similar images**_\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "\n",
    "## Contents\n",
    "\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Setup](#Setup)\n",
    "1. [TensorFlow Model Preparation](#TensorFlow-Model-Preparation)\n",
    "1. [SageMaker Model Hosting](#Hosting-Model)\n",
    "1. [Build a KNN Index in Elasticsearch](#ES-KNN)\n",
    "1. [Evaluate Index Search Results](#Searching-with-ES-k-NN)\n",
    "1. [Extensions](#Extensions)\n",
    "\n",
    "## Background\n",
    "\n",
    "In this notebook, we'll build the core components of a visual image search application. Visual image search is used in interfaces where instead of asking for something by voice or text, you show a photographic example of what you are looking for.\n",
    "\n",
    "One of the core components of visual image search is a convolutional neural net (CNN) model that generates “feature vectors” representing both a query image and the reference item images to be compared against the query. The reference item feature vectors typically are generated offline and must be stored in a database of some sort, so they can be efficiently searched. For small reference item datasets, it is possible to use a brute force search that compares the query against every reference item. However, this is not feasible for large data sets where brute force search would become prohibitively slow. \n",
    "\n",
    "To enable efficient searches for visually similar images, we'll use Amazon SageMaker to generate “feature vectors” from images and use KNN algorithm in Amazon Elasticsearch Service. KNN for Amazon Elasticsearch Service lets you search for points in a vector space and find the \"nearest neighbors\" for those points by Euclidean distance or cosine similarity(default is Euclidean distance). Use cases include recommendations (for example, an \"other songs you might like\" feature in a music application), image recognition, and fraud detection.\n",
    "\n",
    "Here are the steps we'll follow to build the visual image search: After some initial setup, we'll prepare a model using TensorFlow for generating feature vectors, then generate feature vectors of Fashion Images from *__feidegger__*, a *__zalandoresearch__* dataset. Those feature vectors will be imported in Amazon Elasticsearch KNN Index. Next, we'll explore some test image queries, and visualize the results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install tqdm to have progress bar\n",
    "!pip install tqdm\n",
    "\n",
    "#install necessary pkg to make connection with elasticsearch domain\n",
    "!pip install elasticsearch\n",
    "!pip install requests\n",
    "!pip install requests-aws4auth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import re\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "s3_resource = boto3.resource(\"s3\")\n",
    "s3 = boto3.client('s3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfn = boto3.client('cloudformation')\n",
    "\n",
    "def get_cfn_outputs(stackname):\n",
    "    outputs = {}\n",
    "    for output in cfn.describe_stacks(StackName=stackname)['Stacks'][0]['Outputs']:\n",
    "        outputs[output['OutputKey']] = output['OutputValue']\n",
    "    return outputs\n",
    "\n",
    "## Setup variables to use for the rest of the demo\n",
    "cloudformation_stack_name = \"vis-search\"\n",
    "\n",
    "outputs = get_cfn_outputs(cloudformation_stack_name)\n",
    "\n",
    "bucket = outputs['s3BucketTraining']\n",
    "es_host = outputs['esHostName']\n",
    "\n",
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading Zalando Research data\n",
    "\n",
    "The dataset itself consists of 8732 high-resolution images, each depicting a dress from the available on the Zalando shop against a white-background. \n",
    "\n",
    "**Downloading Zalando Research data**: Data originally from here: https://github.com/zalandoresearch/feidegger \n",
    "\n",
    " **Citation:** <br>\n",
    " *@inproceedings{lefakis2018feidegger,* <br>\n",
    " *title={FEIDEGGER: A Multi-modal Corpus of Fashion Images and Descriptions in German},* <br>\n",
    " *author={Lefakis, Leonidas and Akbik, Alan and Vollgraf, Roland},* <br>\n",
    " *booktitle = {{LREC} 2018, 11th Language Resources and Evaluation Conference},* <br>\n",
    " *year      = {2018}* <br>\n",
    " *}*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Data Preparation\n",
    "\n",
    "import os \n",
    "import shutil\n",
    "import json\n",
    "import tqdm\n",
    "import urllib.request\n",
    "from tqdm import notebook\n",
    "from multiprocessing import cpu_count\n",
    "from tqdm.contrib.concurrent import process_map\n",
    "\n",
    "images_path = 'data/feidegger/fashion'\n",
    "filename = 'metadata.json'\n",
    "\n",
    "my_bucket = s3_resource.Bucket(bucket)\n",
    "\n",
    "if not os.path.isdir(images_path):\n",
    "    os.makedirs(images_path)\n",
    "\n",
    "def download_metadata(url):\n",
    "    if not os.path.exists(filename):\n",
    "        urllib.request.urlretrieve(url, filename)\n",
    "        \n",
    "#download metadata.json to local notebook\n",
    "download_metadata('https://raw.githubusercontent.com/zalandoresearch/feidegger/master/data/FEIDEGGER_release_1.1.json')\n",
    "\n",
    "def generate_image_list(filename):\n",
    "    metadata = open(filename,'r')\n",
    "    data = json.load(metadata)\n",
    "    url_lst = []\n",
    "    for i in range(len(data)):\n",
    "        url_lst.append(data[i]['url'])\n",
    "    return url_lst\n",
    "\n",
    "\n",
    "def download_image(url):\n",
    "    urllib.request.urlretrieve(url, images_path + '/' + url.split(\"/\")[-1])\n",
    "                    \n",
    "#generate image list            \n",
    "url_lst = generate_image_list(filename)     \n",
    "\n",
    "workers = 2 * cpu_count()\n",
    "\n",
    "#downloading images to local disk\n",
    "process_map(download_image, url_lst, max_workers=workers)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uploading dataset to S3\n",
    "\n",
    "files_to_upload = []\n",
    "dirName = 'data'\n",
    "for path, subdirs, files in os.walk('./' + dirName):\n",
    "    path = path.replace(\"\\\\\",\"/\")\n",
    "    directory_name = path.replace('./',\"\")\n",
    "    for file in files:\n",
    "        files_to_upload.append({\n",
    "            \"filename\": os.path.join(path, file),\n",
    "            \"key\": directory_name+'/'+file\n",
    "        })\n",
    "        \n",
    "\n",
    "def upload_to_s3(file):\n",
    "        my_bucket.upload_file(file['filename'], file['key'])\n",
    "        \n",
    "#uploading images to s3\n",
    "process_map(upload_to_s3, files_to_upload, max_workers=workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Model Preparation\n",
    "\n",
    "We'll use TensorFlow backend to prepare a model for \"featurizing\" images into feature vectors. TensorFlow has a native Module API, as well as a higher level Keras API. \n",
    "\n",
    "We will start with a pretrained model, avoiding spending time and money training a model from scratch. Accordingly, as a first step in preparing the model, we'll import a pretrained model from Keras application. Researchers have experimented with various pretrained CNN architectures with different numbers of layers, discovering that there are several good possibilities.\n",
    "\n",
    "In this notebook, we'll select a model based on the ResNet architecture, a commonly used choice. Of the various choices for number of layers, ranging from 18 to 152, we'll use 50 layers. This also is a common choice that balances the expressiveness of the resulting feature vectors (embeddings) against computational efficiency (lower number of layers means greater efficiency at the cost of less expressiveness)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import time\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing import image\n",
    "from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input\n",
    "import sagemaker\n",
    "from PIL import Image\n",
    "from sagemaker.tensorflow import TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the channel first for better performance\n",
    "from tensorflow.keras import backend\n",
    "backend.set_image_data_format('channels_first')\n",
    "print(backend.image_data_format())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll get a reference ResNet50 model which is trained on Imagenet dataset to extract the feature without the actual clssifier. More specifically, we'll use that layer to generate a row vector of floating point numbers as an \"embedding\" or representation of the features of the image. We'll also save the model as *SavedModel* format under **export/Servo/1** to serve from SageMaker TensorFlow serving API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import Resnet50 model\n",
    "model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False,input_shape=(3, 224, 224),pooling='avg')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating the directory strcture\n",
    "dirName = 'export/Servo/1'\n",
    "if not os.path.exists(dirName):\n",
    "    os.makedirs(dirName)\n",
    "    print(\"Directory \" , dirName ,  \" Created \")\n",
    "else:    \n",
    "    print(\"Directory \" , dirName ,  \" already exists\")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the model in SavedModel format\n",
    "model.save('./export/Servo/1/', save_format='tf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check the model Signature\n",
    "!saved_model_cli show --dir ./export/Servo/1/ --tag_set serve --signature_def serving_default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SageMaker Model Hosting\n",
    "\n",
    "After saving the feature extractor model we will deploy the model using Sagemaker Tensorflow Serving api which is a flexible, high-performance serving system for machine learning models, designed for production environments.TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data. We will define **inference.py** to customize the input data to TensorFlow serving API. We also need to add **requirements.txt** file for aditional libraby in the tensorflow serving container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check the actual content of inference.py\n",
    "!pygmentize src/inference.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tarfile\n",
    "\n",
    "#zip the model .gz format\n",
    "model_version = '1'\n",
    "export_dir = 'export/Servo/' + model_version\n",
    "with tarfile.open('model.tar.gz', mode='w:gz') as archive:\n",
    "    archive.add('export', recursive=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Upload the model to S3\n",
    "sagemaker_session = sagemaker.Session()\n",
    "inputs = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='model')\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we upload the model to S3 we will use TensorFlow serving container to host the model. We are using ml.p3.16xlarge instance type. You may need to raise support ticket to increase the Service quotas for SageMaker hosting instance type. We will use this endpoint to generate features and import into ElasticSearch. you can also choose small instance such as \"ml.m4.xlarge\" to save cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Deploy the model in Sagemaker Endpoint. This process will take ~10 min.\n",
    "from sagemaker.tensorflow.serving import Model\n",
    "\n",
    "sagemaker_model = Model(entry_point='inference.py', model_data = 's3://' + sagemaker_session.default_bucket() + '/model/model.tar.gz',\n",
    "                                  role = role, framework_version='2.1.0', source_dir='./src' )\n",
    "\n",
    "predictor = sagemaker_model.deploy(initial_instance_count=3, instance_type='ml.m5.xlarge')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the features for a sample image\n",
    "payload = s3.get_object(Bucket=bucket,Key='data/feidegger/fashion/0VB21C000-A11@12.1.jpg')['Body'].read()\n",
    "predictor.content_type = 'application/x-image'\n",
    "predictor.serializer   = None\n",
    "features = predictor.predict(payload)['predictions'][0]\n",
    "\n",
    "features\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a KNN Index in Elasticsearch\n",
    "\n",
    "KNN for Amazon Elasticsearch Service lets you search for points in a vector space and find the \"nearest neighbors\" for those points by Euclidean distance or cosine similarity (default is Euclidean distance). Use cases include recommendations (for example, an \"other songs you might like\" feature in a music application), image recognition, and fraud detection.\n",
    "\n",
    "KNN requires Elasticsearch 7.1 or later. Full documentation for the Elasticsearch feature, including descriptions of settings and statistics, is available in the Open Distro for Elasticsearch documentation. For background information about the k-nearest neighbors algorithm\n",
    "\n",
    "In this step we'll get all the features zalando images and import those features into Elastichseach7.4 domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define some utility function\n",
    "\n",
    "#return all s3 keys\n",
    "def get_all_s3_keys(bucket):\n",
    "    \"\"\"Get a list of all keys in an S3 bucket.\"\"\"    \n",
    "    keys = []\n",
    "\n",
    "    kwargs = {'Bucket': bucket}\n",
    "    while True:\n",
    "        resp = s3.list_objects_v2(**kwargs)\n",
    "        for obj in resp['Contents']:\n",
    "            keys.append('s3://' + bucket + '/' + obj['Key'])\n",
    "\n",
    "        try:\n",
    "            kwargs['ContinuationToken'] = resp['NextContinuationToken']\n",
    "        except KeyError:\n",
    "            break\n",
    "\n",
    "    return keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all the zalando images keys from the bucket make a list\n",
    "s3_uris = get_all_s3_keys(bucket)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a function to extract image features\n",
    "from time import sleep\n",
    "\n",
    "sm_client = boto3.client('sagemaker-runtime')\n",
    "ENDPOINT_NAME = predictor.endpoint\n",
    "\n",
    "def get_predictions(payload):\n",
    "    return sm_client.invoke_endpoint(EndpointName=ENDPOINT_NAME,\n",
    "                                           ContentType='application/x-image',\n",
    "                                           Body=payload)\n",
    "\n",
    "def extract_features(s3_uri):\n",
    "    key = s3_uri.replace(f's3://{bucket}/', '')\n",
    "    payload = s3.get_object(Bucket=bucket,Key=key)['Body'].read()\n",
    "    try:\n",
    "        response = get_predictions(payload)\n",
    "    except:\n",
    "        sleep(0.1)\n",
    "        response = get_predictions(payload)\n",
    "\n",
    "    del payload\n",
    "    response_body = json.loads((response['Body'].read()))\n",
    "    feature_lst = response_body['predictions'][0]\n",
    "    \n",
    "    return s3_uri, feature_lst\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This process cell will take approximately 24-25 minutes on a t3.medium notebook instance\n",
    "# with 3 m5.xlarge SageMaker Hosted Endpoint instances\n",
    "from multiprocessing import cpu_count\n",
    "from tqdm.contrib.concurrent import process_map\n",
    "\n",
    "workers = 2 * cpu_count()\n",
    "result = process_map(extract_features, s3_uris, max_workers=workers)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up the Elasticsearch connection\n",
    "from elasticsearch import Elasticsearch, RequestsHttpConnection\n",
    "from requests_aws4auth import AWS4Auth\n",
    "region = 'us-east-1' # e.g. us-east-1\n",
    "service = 'es'\n",
    "credentials = boto3.Session().get_credentials()\n",
    "awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)\n",
    "\n",
    "es = Elasticsearch(\n",
    "    hosts = [{'host': es_host, 'port': 443}],\n",
    "    http_auth = awsauth,\n",
    "    use_ssl = True,\n",
    "    verify_certs = True,\n",
    "    connection_class = RequestsHttpConnection\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define KNN Elasticsearch index maping\n",
    "knn_index = {\n",
    "    \"settings\": {\n",
    "        \"index.knn\": True\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"zalando_img_vector\": {\n",
    "                \"type\": \"knn_vector\",\n",
    "                \"dimension\": 2048\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating the Elasticsearch index\n",
    "es.indices.create(index=\"idx_zalando\",body=knn_index,ignore=400)\n",
    "es.indices.get(index=\"idx_zalando\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a function to import the feature vectors corrosponds to each S3 URI into Elasticsearch KNN index\n",
    "# This process will take around ~3 min.\n",
    "\n",
    "\n",
    "def es_import(i):\n",
    "    es.index(index='idx_zalando',\n",
    "             body={\"zalando_img_vector\": i[1], \n",
    "                   \"image\": i[0]}\n",
    "            )\n",
    "    \n",
    "process_map(es_import, result, max_workers=workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Index Search Results\n",
    "\n",
    "In this step we will use SageMaker SDK as well as Boto3 SDK to query the Elasticsearch to retrive the nearest neighbours. One thing to mention **zalando** dataset has pretty good similarity with Imagenet dataset. Now if you hav a very domain speific problem then then you need to train that dataset on top of pretrained feature extractor model such as VGG, Resnet, Xeception, Mobilenet etc and bulid a new feature extractor model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define display_image function\n",
    "def display_image(bucket, key):\n",
    "    response = s3.get_object(Bucket=bucket,Key=key)['Body']\n",
    "    img = Image.open(response)\n",
    "    return display(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import random\n",
    "from PIL import Image\n",
    "import io\n",
    "urls = []\n",
    "# yellow pattern dess\n",
    "urls.append('https://fastly.hautelookcdn.com/products/D7242MNR/large/13494318.jpg')\n",
    "# T shirt kind dress\n",
    "urls.append('https://fastly.hautelookcdn.com/products/M2241/large/15658772.jpg')\n",
    "#Dotted pattern dress\n",
    "urls.append('https://fastly.hautelookcdn.com/products/19463M/large/14537545.jpg')\n",
    "\n",
    "img_bytes = requests.get(random.choice(urls)).content\n",
    "query_img = Image.open(io.BytesIO(img_bytes))\n",
    "query_img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### SageMaker SDK Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#SageMaker SDK approach\n",
    "predictor.content_type = 'application/x-image'\n",
    "predictor.serializer   = None\n",
    "features = predictor.predict(img_bytes)['predictions'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "k = 5\n",
    "idx_name = 'idx_zalando'\n",
    "res = es.search(request_timeout=30, index=idx_name,\n",
    "                body={'size': k, \n",
    "                      'query': {'knn': {'zalando_img_vector': {'vector': features, 'k': k}}}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(k):\n",
    "    key = res['hits']['hits'][i]['_source']['image']\n",
    "    key = key.replace(f's3://{bucket}/','')\n",
    "    img = display_image(bucket,key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Boto3 Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = boto3.client('sagemaker-runtime')\n",
    "ENDPOINT_NAME = predictor.endpoint # our endpoint name\n",
    "response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,\n",
    "                                       ContentType='application/x-image',\n",
    "                                       Body=img_bytes)\n",
    "\n",
    "response_body = json.loads((response['Body'].read()))\n",
    "features = response_body['predictions'][0]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "k = 5\n",
    "idx_name = 'idx_zalando'\n",
    "res = es.search(request_timeout=30, index=idx_name,\n",
    "                body={'size': k, \n",
    "                      'query': {'knn': {'zalando_img_vector': {'vector': features, 'k': k}}}})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(k):\n",
    "    key = res['hits']['hits'][i]['_source']['image']\n",
    "    key = key.replace(f's3://{bucket}/','')\n",
    "    img = display_image (bucket,key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying a full-stack visual search application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_resource.Object(bucket, 'backend/template.yaml').upload_file('./backend/template.yaml', ExtraArgs={'ACL':'public-read'})\n",
    "\n",
    "\n",
    "sam_template_url = f'https://{bucket}.s3.amazonaws.com/backend/template.yaml'\n",
    "\n",
    "# Generate the CloudFormation Quick Create Link\n",
    "\n",
    "print(\"Click the URL below to create the backend API for visual search:\\n\")\n",
    "print((\n",
    "    'https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review'\n",
    "    f'?templateURL={sam_template_url}'\n",
    "    '&stackName=vis-search-api'\n",
    "    f'&param_BucketName={outputs[\"s3BucketTraining\"]}'\n",
    "    f'&param_DomainName={outputs[\"esDomainName\"]}'\n",
    "    f'&param_ElasticSearchURL={outputs[\"esHostName\"]}'\n",
    "    f'&param_SagemakerEndpoint={predictor.endpoint}'\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a working Amazon SageMaker endpoint for extracting image features and a KNN index on Elasticsearch, you are ready to build a real-world full-stack ML-powered web app. The SAM template you just created will deploy an Amazon API Gateway and AWS Lambda function. The Lambda function runs your code in response to HTTP requests that are sent to the API Gateway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Review the content of the Lambda function code.\n",
    "!pygmentize backend/lambda/app.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Once the CloudFormation Stack shows CREATE_COMPLETE, proceed to this cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the REST endpoint for the search API to a config file, to be used by the frontend build\n",
    "\n",
    "import json\n",
    "api_endpoint = get_cfn_outputs('vis-search-api')['ImageSimilarityApi']\n",
    "\n",
    "with open('./frontend/src/config/config.json', 'w') as outfile:\n",
    "    json.dump({'apiEndpoint': api_endpoint}, outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Deploy frontend services"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add NPM to the path so we can assemble the web frontend from our notebook code\n",
    "\n",
    "from os import environ\n",
    "\n",
    "npm_path = ':/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin'\n",
    "\n",
    "if npm_path not in environ['PATH']:\n",
    "    ADD_NPM_PATH = environ['PATH']\n",
    "    ADD_NPM_PATH = ADD_NPM_PATH + npm_path\n",
    "else:\n",
    "    ADD_NPM_PATH = environ['PATH']\n",
    "    \n",
    "%set_env PATH=$ADD_NPM_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ./frontend/\n",
    "\n",
    "!npm install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!npm run-script build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hosting_bucket = f\"s3://{outputs['s3BucketHostingBucketName']}\"\n",
    "\n",
    "!aws s3 sync ./build/ $hosting_bucket --acl public-read"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Browse your frontend service, and upload an image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Click the URL below:\\n')\n",
    "print(outputs['S3BucketSecureURL'] + '/index.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the following page:\n",
    "\n",
    "![Website](pi3small.png)\n",
    "\n",
    "On the website, try pasting the following URL in the URL text field.\n",
    "\n",
    "`https://i4.ztat.net/large/VE/12/1C/14/8K/12/VE121C148-K12@10.jpg`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extensions\n",
    "\n",
    "We have used pretrained Resnet50 model which is trained on Imagenet dataset. Now based on your use-case you can fine tune any pre-trained models, such as VGG, Inception, and MobileNet with your own dataset and host the model in Amazon SageMaker.\n",
    "\n",
    "You can also use Amazon SageMaker Batch transform job to have a bulk feaures extracted from your stored S3 images and then you can use AWS Glue to import that data into Elasticeearch domain.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup\n",
    "\n",
    "Make sure that you stop the notebook instance, delete the Amazon SageMaker endpoint and delete the Elasticsearch domain to prevent any additional charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the endpoint\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Empty S3 Contents\n",
    "training_bucket_resource = s3_resource.Bucket(bucket)\n",
    "training_bucket_resource.objects.all().delete()\n",
    "\n",
    "hosting_bucket_resource = s3_resource.Bucket(outputs['s3BucketHostingBucketName'])\n",
    "hosting_bucket_resource.objects.all().delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}