cwpk-23-text-searching-kbpedia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#23: Text Searching KBpedia\n",
    "==============================\n",
    "\n",
    "\n",
    "Using the Direct Approach with Owlready2\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In this installment of the [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series, we explore ways to directly search knowledge graph text from within the [owlready2](http://www.lesfleursdunormal.fr/static/informatique/owlready/index_en.html) [API](https://en.wikipedia.org/wiki/Application_programming_interface). We first introduced this topic in <strong>[CWPK #19](https://www.mkbergman.com/2349/cwpk-19-exploring-the-api-to-owl/)</strong>; we explain further some of the nuances here.\n",
    "\n",
    "Recall that owlready2 uses its own local datastore, [SQLite](https://en.wikipedia.org/wiki/SQLite), for storing its knowledge graphs. Besides the search functionality added in Owlready2, we will also be taking advantage of the full-text search (FTS) functionality within SQLite.\n",
    "\n",
    "### Load Full Knowledge Graph\n",
    "To get started, we again load our working knowledge graph. In this instance we will use the full KBpedia knowledge graph, <code>kbpedia_reference_concepts.owl</code>, because it has a richer set of contents.\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service (the 'raw' version) or local files. The example below is based on using local files (though replace with your own local directory specification). If loading from MyBinder, use this <a href=\"https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl\">address for <code>kbpedia_reference_concepts.owl</code></a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "main = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'\n",
    "# main = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl'\n",
    "skos_file = 'http://www.w3.org/2004/02/skos/core' \n",
    "kko_file = 'C:/1-PythonProjects/owlready2/kg/kko.owl'\n",
    "# kko_file = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl'\n",
    "\n",
    "from owlready2 import *\n",
    "world = World()\n",
    "kb = world.get_ontology(main).load()\n",
    "rc = kb.get_namespace('http://kbpedia.org/kko/rc/')\n",
    "\n",
    "skos = world.get_ontology(skos_file).load()\n",
    "kb.imported_ontologies.append(skos)\n",
    "\n",
    "kko = world.get_ontology(kko_file).load()\n",
    "kb.imported_ontologies.append(kko)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To execute the load, pick <code>shift+enter</code> to execute the cell contents, or pick Run from the main menu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides changing our absolute file input, note we have added another scoping assignment <code>world</code> to our load. <code>world</code> is a reserved keyword in Owlready2 that encompasses the SQLite storage space used by Owlready2. Note we assign all of our ontologies (knowledge graphs) to this namespace so that we may invoke some of the FTS functionality later in this installment.\n",
    "\n",
    "### Basic Search Functions\n",
    "\n",
    "As the [owlready2 documentation](xxx) explains, it contains some pre-loaded search capabilities that can be performed with the <code>.search()</code> query method. This method can accept one or several keyword arguments:\n",
    "\n",
    "- <code>iri</code> - for searching entities by their full IRIs\n",
    "- <code>type</code> - for searching instances for a given class\n",
    "- <code>subclass_of</code> - for searching subclasses of a given class\n",
    "- <code>is_a</code> - for searching both instances and subclasses of a given class, or object, data or annotation property name.\n",
    "\n",
    "Special arguments that may be added to these arguments are:\n",
    "\n",
    "- <code>_use_str_as_loc_str</code> - whether to treats plain Python strings as strings in any language (default is True)\n",
    "- <code>_case_sensitive</code> - whether to take lower/upper case into consideration (default is True).\n",
    "\n",
    "Our search queries may accept quoted phrases and prefix or suffix wildcards (*). Let's look at some examples combining these arguments and methods. Our first one is similar to what we presented in <strong>CWPK #19</strong>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "world.search(iri = \"*luggage*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice our result here is an empty set, in other words, no matches. Yet we know there are IRIs in KBpedia that include the term 'luggage'. We suspect the reason for not seeing a match is that the term might start with upper case in our IRIs. We will set the case sensitivity argument to false and try again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.search(iri = \"*luggage*\", _case_sensitive = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We are now seeing the results we expected.\n",
    "\n",
    "Note in the query above that we used the wildcard (*) to allow for either prefix or suffix matches. As you can see from the results above, most of the search references match the interior part of the IRI string.\n",
    "\n",
    "The <code>iri</code> argument takes a search string as its assignment. The other three keyword assignments noted above take an object name, as this next example shows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.search(subclass_of=rc.Mammal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a tremendous number of matches on this query, so much so that I cleared away the current cell output (via Cell &rarr; Current Outputs &rarr; Clear, when highlighting this cell). To winnow this results set further, we can combine search terms as the next example shows. We will add to our initial search a string search in the IRIs for which prior results might represent 'Bats':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[rc.Bat-Mammal, rc.SacWingedBat, rc.BulldogBat, rc.FreeTailedBat, rc.HorseshoeBat, rc.SchreibersBat, rc.WesternSuckerFootedBat, rc.AfricanLongFingeredBat, rc.AfricanYellowBat, rc.AllensYellowBat, rc.AsianPartiColoredBat, rc.DaubentonsBat, rc.EasternRedBat, rc.GreatEveningBat, rc.GreaterTubeNosedBat, rc.GreyLongEaredBat, rc.HawaiianHoaryBat, rc.KobayashisBat, rc.LesserYellowBat, rc.LittleTubeNosedBat, rc.NewGuineaBigEaredBat, rc.NewZealandLongTailedBat, rc.NorthernLongEaredBat, rc.PallidBat, rc.SilverHairedBat, rc.SpottedBat, rc.AfricanSheathTailedBat, rc.AmazonianSacWingedBat, rc.BeccarisSheathTailedBat, rc.ChestnutSacWingedBat, rc.DarkSheathTailedBat, rc.EcuadorianSacWingedBat, rc.EgyptianTombBat, rc.FrostedSacWingedBat, rc.GraySacWingedBat, rc.GreaterSacWingedBat, rc.GreaterSheathTailedBat, rc.GreenhallsDogFacedBat, rc.HamiltonsTombBat, rc.HildegardesTombBat, rc.LargeEaredSheath-TailedBat, rc.LesserSacWingedBat, rc.LesserSheathTailedBat, rc.MauritianTombBat, rc.NorthernGhostBat, rc.PacificSheathTailedBat, rc.PelsPouchedBat, rc.PeterssSheathTailedBat, rc.ProboscisBat, rc.RaffraysSheathTailedBat, rc.SerisSheathtailBat, rc.SeychellesSheathTailedBat, rc.ShaggyBat, rc.ShortEaredBat, rc.SmallAsianSheathTailedBat, rc.TheobaldsTombBat, rc.ThomassSacWingedBat, rc.TroughtonsPouchedBat, rc.AntilleanFruitEatingBat, rc.BidentateYellowEaredBat, rc.BigEaredWoolyBat, rc.CommonVampireBat, rc.HairyLeggedVampireBat, rc.HonduranWhiteBat, rc.LesserLongNosedBat, rc.MexicanLongNosedBat, rc.SpectralBat, rc.VampireBat, rc.WhiteWingedVampireBat, rc.GreaterBulldogBat, rc.LesserBulldogBat, rc.BigCrestedMastiffBat, rc.BigFreeTailedBat, rc.BlackBonnetedBat, rc.BroadEaredBat, rc.EuropeanFreeTailedBat, rc.GallaghersFreeTailedBat, rc.IncanLittleMastiffBat, rc.LittleGoblinBat, rc.MexicanFreeTailedBat, rc.NatalFreeTailedBat, rc.PetersWrinkleLippedBat, rc.SumatranMastiffBat, rc.WroughtonsFreeTailedBat, rc.LesserFalseVampireBat, rc.YellowWingedBat, rc.BigNakedBackedBat, rc.ParnellsMustachedBat, rc.NewZealandGreaterShortTailedBat, rc.NewZealandLesserShortTailedBat, rc.BatesSlitFacedBat, rc.LargeSlitFacedBat, rc.DayakFruitBat, rc.LittleMarianaFruitBat, rc.LivingstonesFruitBat, rc.MarianaFruitBat, rc.PetersDiskWingedBat, rc.Bat-earedFox, rc.AboBat, rc.JapaneseHouseBat, rc.BigBrownBat, rc.NorthernBat, rc.GrayBat, rc.GreaterMouseEaredBat, rc.HodgsonsBat, rc.IkonnikovsBat, rc.IndianaBat, rc.LittleBrownBat, rc.NatterersBat, rc.GreaterNoctuleBat, rc.VirginiaBigEaredBat, rc.GreaterHorseshoeBat, rc.LamottesRoundleafBat, rc.LesserHorseshoeBat, rc.MalayanRoundleafBat, rc.VietnamLeafNosedBat, rc.PersianTridentBat, rc.BondaMastiffBat, rc.VelvetyFreeTailedBat, rc.WesternMastiffBat, rc.MexicanFunnelEaredBat, rc.AndersensFruitEatingBat, rc.JamaicanFruitBat]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "world.search(subclass_of=rc.Mammal, iri = \"*Bat*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we get a large number of results. There are clearly many mammals and bats within the KBpedia reference graph!\n",
    "\n",
    "Per the listing above, there are a number of these pre-configured search arguments directly available through Owlready2.\n",
    "\n",
    "### Full Text Search\n",
    "We can also instruct the FTS system in SQLite that we want to index still additional fields. Since we are interested in  a term we know occurs in KBpedia's annotations relating some reference concepts to the UN standard products and services codes ([UNSPSC](https://en.wikipedia.org/wiki/UNSPSC)) we try that search directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "world.search(entered = \"*UNSPSC*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, this tells us there are no results. We must be missing an indexed field. So, let's instruct the system to add indexing to the <code>definition</code> property where we suspect the reference may occur. We do so using the <code>.append</code> method to add a new field for our RC definitions (<code>skos.definition</code>) to the available FTS index structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.full_text_search_properties.append(skos.definition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is just a simple assignment, when we Run the cell we get no results output. \n",
    "\n",
    "However, that assignment now allows us to invoke the internal FTS (full-text search) argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.search(definition = FTS(\"UNSPSC*\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get an 'operational error' that means you did not Run the <code>.append</code> instruction above.\n",
    "\n",
    "Like some of the other listings, this command results in a very large number of results, a couple of which are warnings we can ignore, so we again Clear the Cell. We can get a smaller listing with another keyword search, this time for the wildcarded 'gear*' search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[rc.undercarriage, rc.number-of-forward-gears, rc.vehicle-transmission, rc.AutomaticTransmission, rc.BearingBushingWheelGear, rc.BevelGear, rc.Bicycle-MultiGear, rc.BoeingAH-64Apache, rc.BugattiVeyron, rc.ChildrensWebSite, rc.CombatSportsEvent, rc.Commercialism, rc.CyclingClothing, rc.Device-FunctionallyDefective, rc.FirstNorthAmericansNovels, rc.Fishery, rc.FreeDiving, rc.Game-EquipmentSet, rc.Gear, rc.GearManufacturingMachine, rc.Gearing-Mechanical, rc.GearlessElectricDrive, rc.Goggles, rc.Harness-Animal, rc.Helmet, rc.IlyushinIl-30, rc.LandingGearAssembly, rc.MachineProtocol, rc.Mechanism-Technology, rc.Overdrive-Mechanics, rc.PinionGear, rc.ProtectiveEquipment-Human, rc.ProtectiveGear, rc.ScubaGear, rc.ScubaSnorkelingGear, rc.ShockAndAwe-MilitaryTactic, rc.Supercharger, rc.TeacherTrainingProgram, rc.Trek, rc.Wheel, rc.WildernessBackpacking, rc.WormGear]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "world.search(definition = FTS(\"gear*\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice in this search that we are able to use the suffix wildcard (*) character. However, unlike the standard OWLready2 search, we are not able to use a wildcard (*) prefix search.\n",
    "\n",
    "Since we have added a new indexed search table to our system, we may want to retain this capability. So, we decide to save the entire graph to the database, as the last example shows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.set_backend(filename = 'cwpk-23-text-searching-kbpedia.db', exclusive = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This now means our database has been saved persistently to disk.\n",
    "\n",
    "If you run this multiple times you may get an operational error since you have already set the backend filename.\n",
    "\n",
    "We can then <code>.save()</code> our work and exit the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "world.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "Here is additional information on the system's text searching capabilities:\n",
    "\n",
    "- Standard [owlready2 search options](https://owlready2.readthedocs.io/en/latest/onto.html#simple-queries)\n",
    "- [FTS search options](https://owlready2.readthedocs.io/en/latest/annotations.html#full-text-search-fts).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}