binder/classify-img-with-iiif-and-watson.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This Python 3 notebook extracts images of a Gallica document (using the IIIF protocol), and then applies an IBM Watson classification model to the images**\n",
    "1. Extract the document bibliographical metadata from the Gallica OAI-PMH repository\n",
    "2. Extract the document technical image metadata from its IIIF manifest, and then the images\n",
    "3. Classify the images with a Watson Cloud Vision model (the model must be available)\n",
    "\n",
    "<i>Prerequisites</i>:\n",
    "- a pretrained  Watson Sudio classification model identified with its ID\n",
    "- a Watson API key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert here the Gallica document ID you want to process\n",
    "doc_ID = '12148/btv1b10336854c'\n",
    "# CSV export\n",
    "output = \"OUT_csv\"\n",
    "# get doc_max first images from the document\n",
    "doc_max = 2\n",
    "# IIIF export factor (%)\n",
    "doc_export_factor = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "print(\"Python version\")\n",
    "print (sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. PyGallica (https://github.com/ian-nai/PyGallica) is used to access the Gallica OAI and get some bibliographic metadata. The call returns metadata as a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we import the Document class from the PyGallica package (https://github.com/ian-nai/PyGallica)\n",
    "from document_api import Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# we get the document metadata with the Gallica OAI API wrapped within the PyGallica package\n",
    "#json_dict4doc = Document.OAI(doc_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do whatever you want with the document bibliographic metadata\n",
    "# get the title\n",
    "#print json_dict4doc['results']['title']\n",
    "# get the Dublin Core medatada\n",
    "#print json_dict4doc['results']['notice']['record']['metadata']['oai_dc:dc']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. We ask for the document IIIF manifest to then have access to the images files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we build the IIIF URL\n",
    "import requests\n",
    "METADATA_BASEURL = 'https://gallica.bnf.fr/iiif/ark:/'\n",
    "req_url = \"\".join([METADATA_BASEURL, doc_ID, '/manifest.json'])\n",
    "print (\"... getting the IIIF manifest\",req_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we ask for the manifest. The call returns a dictionary\n",
    "r = requests.get(req_url)\n",
    "r.raise_for_status()\n",
    "json_4img = r.json()\n",
    "print (json_4img.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the sequence of images metadata. It's a list\n",
    "sequences = json_4img.get('sequences')\n",
    "# get the canvases, first element of the list. Its a dict\n",
    "canvases = sequences[0]\n",
    "print (canvases.keys())\n",
    "# parse each canvas data for each image\n",
    "# each canvas has these keys: [u'height', u'width', u'@type', u'images', u'label', u'@id', u'thumbnail']\n",
    "n_images = 0\n",
    "# the array of URLs we're going to build\n",
    "urlsIIIF = [] \n",
    "print (\"... getting image metadata from the IIIF manifest\")\n",
    "for c in canvases.get('canvases'): \n",
    "    n_images += 1\n",
    "    print (\"    label:\",c.get('label'),\" width:\",c.get('width'), \" height:\",c.get('height'))\n",
    "    # we also get a Gallica thumbnail (it's not a IIIF image)\n",
    "    thumbnail = c.get('thumbnail')\n",
    "    urlThumbnail = thumbnail.get('@id')\n",
    "    # we build the IIIF URL. We ask for the full image with a size factor of docExportFactor\n",
    "    urlIIIF = \"\".join([doc_ID,'/f',str(n_images)]), 'full', \"\".join(['pct:',str(doc_export_factor)]), '0', 'native', 'jpg'\n",
    "    urlsIIIF.append(urlIIIF)\n",
    "    if n_images >= doc_max:\n",
    "        break\n",
    "        \n",
    "print (\"--------------\")\n",
    "print (f\"... we get {doc_max} images on {len(canvases.get('canvases'))}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we'd like to get the image files with the IIIF Image API (PyGallica package again)\n",
    "from iiif_api import IIIF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now loading the images\n",
    "# the files are stored in the . folder\n",
    "[IIIF.iiif(u[0],u[1],u[2],u[3],u[4],u[5]) for u in urlsIIIF]\n",
    "\n",
    "\"\"\"\n",
    "# get the image files #x to #y (we only process y-x images)\n",
    "for i in range(8, 9):\n",
    "    print (\"--- getting image...\")\n",
    "    # we build the IIIF URL. We ask for the full image with a size factor of docExportFactor\n",
    "    IIIF.iiif(\"\".join([docID,'/f',str(i)]), 'full', \"\".join(['pct:',str(doc_export_factor)]), '0', 'native', 'jpg')\n",
    "\"\"\"\n",
    "print (\"... done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Now we have to call the Watson classification model on the local image files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import PIL.Image\n",
    "from IPython.display import Image, display\n",
    "import os, fnmatch\n",
    "\n",
    "#######################\n",
    "outputDir = os.path.realpath(output)\n",
    "if not os.path.isdir(outputDir):\n",
    "    print(f\"\\n  Output .csv directory {outputDir} does not exist!\")\n",
    "    os.mkdir( outputDir);\n",
    "else:\n",
    "    print (f\"\\n... CSV files will be saved to {outputDir}\")\n",
    "    \n",
    "# Watson parameters:\n",
    "WATSON_BASEURL = 'https://gateway.watsonplatform.net/visual-recognition/api/v3/classify?version=2018-03-19'\n",
    "WATSON_VERSION = (('version', '2018-03-19'),)\n",
    "# insert your Watson key here\n",
    "WATSON_KEY = '***' \n",
    "# insert your Watson visual recognition model ID here\n",
    "WATSON_MODEL = 'DefaultCustomModel_1457318034'\n",
    "\n",
    "def process_image(file_name):\n",
    "    # we use the requests package\n",
    "    files = {\n",
    "        'images_file': (file_name, open(file_name, 'rb')),\n",
    "        'classifier_ids': (None, WATSON_MODEL),\n",
    "    }\n",
    "    # calling the Watson API \n",
    "    return requests.post('https://gateway.watsonplatform.net/visual-recognition/api/v3/classify',params=WATSON_VERSION, files=files, auth=('apikey', '2DZdFVOxmhJnQIq3MOQsUOg-7RqARAWa35q-Gh31NnSK'))\n",
    "\n",
    "def process_json(watson_json):\n",
    "    images = watson_json.get('images')\n",
    "    watson_class = images[0].get('classifiers')[0].get('classes')[0].get('class')\n",
    "    watson_score = images[0].get('classifiers')[0].get('classes')[0].get('score')\n",
    "    print (\"    -> classification:\",watson_class)\n",
    "    print (\"    -> confidence score:\",watson_score)\n",
    "    return(watson_class,watson_score)\n",
    "    \n",
    "# writing the results in out_path\n",
    "out_path = os.path.join(output, \"classifications.csv\" )\n",
    "out_file = open(out_path,\"w\")\n",
    "\n",
    "# first we read the images\n",
    "# the images have been stored in a folder based on the document ID\n",
    "# like 12148/btv1b103365619\n",
    "entries = fnmatch.filter(os.listdir(doc_ID), '*.jpg')\n",
    "for file in entries:\n",
    "    file_name = \"\".join([doc_ID,\"/\",file])\n",
    "    print (\"--- infering image \",file_name,\" ...\")\n",
    "    # we display the image\n",
    "    #img = Image.open(file_name)\n",
    "    #img.show() \n",
    "    display(Image(filename=file_name))\n",
    "    # calling the Watson API \n",
    "    watson_json = process_image(file_name).json()\n",
    "    # Watson returns a JSON with classification and confidence score informations\n",
    "    #print(json.dumps(json_watson, sort_keys=True, indent=4))  \n",
    "    result = process_json(watson_json)\n",
    "    predicted_class = result[0]\n",
    "    predicted_class = predicted_class[:-3]\n",
    "    # write in file\n",
    "    print (\"%s\\t%s\" % (file_name, predicted_class.lower()), file=out_file)\n",
    "    \n",
    "out_file.close()\n",
    "print (\"\\n... writing classification data in\",output)\n",
    "exit(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. We evaluate the performances relatively to a ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GT_folder = \"GT\"\n",
    "labels = []\n",
    "predictions = []\n",
    "predictions = []\n",
    "trues = []\n",
    "\n",
    "# infering the GT data\n",
    "folders = fnmatch.filter(os.listdir(GT_folder), '*')\n",
    "for f in folders:\n",
    "    folder_name = \"\".join([GT_folder,\"/\",f])\n",
    "    if os.path.isdir(folder_name):\n",
    "        print (\"\\n--- GT class \",f,\" ...\")\n",
    "        labels.append(f)\n",
    "        images = fnmatch.filter(os.listdir(folder_name), '*.jpg')\n",
    "        for i in images:\n",
    "            trues.append(f)\n",
    "            file_name = \"\".join([GT_folder,\"/\",f,\"/\",i])\n",
    "            print (\"    infering image:\",file_name,\" ...\")\n",
    "            watson_json = process_image(file_name).json()\n",
    "            result = process_json(watson_json)\n",
    "            predicted_class = result[0]\n",
    "            predicted_class = predicted_class[:-3] # remove the \"_30\" (class names in the Watson model)\n",
    "            predictions.append(predicted_class.lower())\n",
    "print (\"classes:\",labels)\n",
    "print (\"GT:\", trues)\n",
    "print (\"predictions:\",predictions)\n",
    "\n",
    "# computing the preformances: confusion matrix, recall, precision\n",
    "import seaborn\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def plot_confusion_matrix(data, labels, output_filename):\n",
    "    \"\"\"Plot confusion matrix using heatmap.\n",
    " \n",
    "    Args:\n",
    "        data (list of list): List of lists with confusion matrix data.\n",
    "        labels (list): Labels which will be plotted across x and y axis.\n",
    "        output_filename (str): Path to output file.\n",
    " \n",
    "    \"\"\"\n",
    "    seaborn.set(color_codes=True)\n",
    "    plt.figure(1, figsize=(9, 6))\n",
    " \n",
    "    plt.title(\"Confusion Matrix\")\n",
    " \n",
    "    seaborn.set(font_scale=1.4)\n",
    "    ax = seaborn.heatmap(data, annot=True, cmap=\"YlGnBu\", cbar_kws={'label': 'Scale'})\n",
    " \n",
    "    ax.set_xticklabels(labels)\n",
    "    ax.set_yticklabels(labels)\n",
    " \n",
    "    ax.set(ylabel=\"True Label\", xlabel=\"Predicted Label\")\n",
    " \n",
    "    plt.savefig(output_filename, bbox_inches='tight', dpi=120)\n",
    "    plt.close()\n",
    " \n",
    " \n",
    "# create confusion matrix\n",
    "matrix = confusion_matrix(trues, predictions, labels=labels)\n",
    "print (matrix)\n",
    "\n",
    "# compute the recall score\n",
    "# https://simonhessner.de/why-are-precision-recall-and-f1-score-equal-when-using-micro-averaging-in-a-multi-class-problem/\n",
    "# 'micro': Calculate metrics globally by counting the total true positives, false negatives and false positives.\n",
    "# 'macro': Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.\n",
    "print (\"\\nrecall score:\", recall_score(trues, predictions, labels=labels, average='micro'))\n",
    "print (\"precision score:\", precision_score(trues, predictions, labels=labels, average='micro'))\n",
    "print (\"F1 mesure:\", f1_score(trues, predictions, labels=labels, average='micro'))\n",
    "       \n",
    "# draw the matrix\n",
    "plot_confusion_matrix(matrix, labels, \"confusion_matrix.png\")\n",
    "display(Image(filename=\"confusion_matrix.png\"))\n",
    "exit(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. We could do the same on a IIIF image URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "# Wellcome collection\n",
    "iiifURL = \"https://iiif.wellcomecollection.org/image/L0009407.jpg/1,1,1568,1213/1000,/0/default.jpg\"\n",
    "# Gallica\n",
    "#iiifURL = \"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k4628326j/f1/4317.695641814265,2899.28514719721,1006.9642711679644,774.944853848157/217,167/0/native.jpg\"\n",
    "CURL_URL = (('url', iiifURL),)\n",
    "WATSON_CLASSIFIER = (('classifier_ids', WATSON_MODEL),)\n",
    "curlParams = {\n",
    "        'url': (None, iiifURL),\n",
    "        'classifier_ids': (None, WATSON_MODEL),\n",
    "        'version': (None, '2018-03-19')\n",
    "    }\n",
    "print (\"--- infering image \",iiifURL,\" ...\")\n",
    "img = PIL.Image.open(requests.get(iiifURL, stream=True).raw)\n",
    "#img.show() \n",
    "display(img)\n",
    "# call to the Watson API \n",
    "response = requests.post('https://gateway.watsonplatform.net/visual-recognition/api/v3/classify',params=curlParams, auth=('apikey', '2DZdFVOxmhJnQIq3MOQsUOg-7RqARAWa35q-Gh31NnSK'))\n",
    "json_watson = response.json()\n",
    "print(json.dumps(json_watson, sort_keys=True, indent=4)) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}