cwpk-39-structural-ingest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#39: I/O and Structural Ingest\n",
    "=======================================\n",
    "\n",
    "Builds Are a More Complicated Workflow than Extractions\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In installment [**CWPK #37**](https://www.mkbergman.com/2376/cwpk-37-organizing-the-code-base/) to this [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series, we looked at [roundtripping](https://en.wikipedia.org/wiki/Round-trip_format_conversion) information flows and their relation to versions and broad directory structure. From a strict order-of-execution basis, one of the first steps in the build process is to check and vet the input files to make sure they are clean and conformant. Let's keep that thought in mind, and it is a topic we will address in **CWPK #45**. But we are going to skip over that topic for now so that we do not add complexity to figuring out the basis of what we need from a backbone standpoint of a build.\n",
    "\n",
    "By \"build\" I have been using a single term to skirt around a number of issues (or options). We can \"build\" our system from scratch at all times, which is the simplest conceptual one in that we go from A to Z, processing every necessary point in between. But, sometimes a \"build\" is not a wholesale replacement, but an incremental one. It might be incremental because we are simply adding, say, a single new typology or doing other interim updates. The routines or methods we need to write should accommodate these real-world use cases.\n",
    "\n",
    "We also have the question of vetting our builds to ensure they are syntactically and logically coherent. The actual process of completing a vetted, 'successful' build may require multiple iterations and successively more refined tweaks to the input specifications in order to get our knowledge graphs to a point of 'acceptance'. These last iterating steps of successive refinements all follow the same \"build\" steps, but ones which involve fewer and fewer fixes until 'acceptance' when the knowledge graph is deemed ready for public release. In my own experience following these general build steps over nearly a decade now, there may be tens or more \"builds\" necessary to bring a new version to public release. These steps may not be needed in your own circumstance, but a useful generic build process should anticipate them.\n",
    "\n",
    "Further, as we explore these issues in some depth, we will find weaknesses or problems in our earlier extraction flows. We still want to find generic patterns and routines, but as we add these ones related to \"builds\" that will also cause some reachback into the assumptions and approaches we earlier used for extractions ([**CWPK #29**](https://www.mkbergman.com/2364/cwpk-29-extracting-object-and-data-properties/) to [**CWPK #35**](https://www.mkbergman.com/2372/cwpk-35-a-python-module-part-iii-custom-extractions-and-finish-packaging/)). This installment thus begins with input-output (I/O) considerations from this \"build\" perspective."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic I/O Considerations\n",
    "It seems like I spend much time in this series on code and directory architecting. I'm sure one reason is that it is easier for me to write than code! But, seriously, it is also the case that thinking through use cases or trying to capture prior workflows is the best way I know to write responsive code. So, in the case of our first leg regarding extraction, we had a fairly simple task: we had information in our knowledge graphs that we needed to get out in a usable file format for external applications (and roundtripping). If our file names and directory locations were not exactly correct, no big deal: We can easily manipulate these flat-text files and move them to other places as needed.\n",
    "\n",
    "That is not so with the builds. First, we can only ingest our external files used in a build that can be read and \"understood\" by our knowledge graph. Second, as I touched on above, sometimes these builds may be total (or 'full' or 'start'), sometimes they may be incremental (or 'fix'). We want generic input-and-output routines to reflect these differences.\n",
    "\n",
    "In a 'full' build scenario, we need to start with a core, bootstrap, skeletal ontology to which we add concepts (classes) and predicates (properties) until the structure is complete (or nearly so), and then to add annotations to all of that input. In a 'fix' build scenario, we need to start with an already constructed 'full' graph, and then make modifications (updates, deletes, additions) to it. How much cleaning or logical testing we may do may vary by these steps. We also need to invoke and write files to differing locations to embrace these options.\n",
    "\n",
    "To make this complexity a bit simpler, like we have done before, we will make a couple of simplifying choices. First, we will use a directory where all build input files reside, which we call <code>build_ins</code>. This directory is the location where we first put the files extracted from a prior version to be used as the starting basis for the new version (see *Figure 2* in [**CWPK #37**](https://www.mkbergman.com/2376/cwpk-37-organizing-the-code-base/)). It is also the directory where we place our starting ontology files, the <code>stubs</code>, that bootstrap the locations for new properties and classes to be added. We also place our <code>fixes</code> inputs into this directory.\n",
    "\n",
    "Second, the result of our various build steps will generally be placed into a single sub-directory, the <code>targets</code> directory. This directory is the source for all completed builds used for analysis and extractions for external uses and new builds. It is also the source of the knowledge graph input when we are in an incremental update or 'fix' mode, since we desire to modify the current build in-progress, not always start from scratch. The <code>targets</code> directory is also the appropriate location for logging, statistics, and working 'scratchpad' subdirectories while we are working on a given build. \n",
    "\n",
    "To this structure I also add a <code>sandbox</code> directory for experiments, etc., that do not fall within a conventional build paradigm. The <code>sandbox</code> material can either be total scratch or copied manually to other locations if there is some other value.\n",
    "\n",
    "Please see *Figure 2* in [**CWPK #37**](https://www.mkbergman.com/2376/cwpk-37-organizing-the-code-base/) to see the complete enumeration of these directory structures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic I/O Routines\n",
    "Similar to what we did with the extraction side of the roundtrip, we will begin our structural builds (and the annotation ones two installments hence) in the interactive format of Jupyter Notebook. We will be able to progress cell-by-cell Running these or invoking them with the <code>shift+enter</code> convention. After our cleaning routines in **CWPK #45**, we will then be able to embed these interactive routines into <code>build</code> and <code>clean</code> modules in **CWPK #47** as part of the *cowpoke* package.\n",
    "\n",
    "From the get-go with the <code>build</code> module we need to have a more flexible load routine for *cowpoke* that enables us to specify different sources and targets for the specific build, the inputs-outputs, or I/O. We had already discovered in the extraction routines that we needed to bring three ontologies into our project namespace, [KKO](https://kbpedia.org/docs/kko-upper-structure/), the reference concepts of [KBpedia](https://kbpedia.org/), and [SKOS](https://en.wikipedia.org/wiki/Simple_Knowledge_Organization_System). We may also need to differentiate 'start' v 'fix' wrinkles in our builds. That leads to three different combinations of source and target: 'standard' (same as 'fixes'), 'start', and our optional 'sandbox') for our basic \"build\" I/O:\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service or local files. The example below is based on using local files, which given the complexity of the routines that are emerging, is probably your better choice. Make sure to modify the URIs for your local directories.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from owlready2 import * \n",
    "from cowpoke.config import *\n",
    "# from cowpoke.__main__ import *\n",
    "import csv                                                # we import all modules used in subsequent steps\n",
    "\n",
    "world = World()\n",
    "\n",
    "kko = []\n",
    "kb = []\n",
    "rc = []\n",
    "core = []\n",
    "skos = []\n",
    "kb_src = build_deck.get('kb_src')                         # we get the build setting from config.py\n",
    "\n",
    "if kb_src is None:\n",
    "    kb_src = 'standard'\n",
    "if kb_src == 'sandbox':\n",
    "    kbpedia = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'\n",
    "    kko_file = 'C:/1-PythonProjects/kbpedia/sandbox/kko.owl'\n",
    "elif kb_src == 'standard':\n",
    "    kbpedia = 'C:/1-PythonProjects/kbpedia/v300/targets/ontologies/kbpedia_reference_concepts.owl'\n",
    "    kko_file = 'C:/1-PythonProjects/kbpedia/v300/build_ins/stubs/kko.owl'\n",
    "elif kb_src == 'start':\n",
    "    kbpedia = 'C:/1-PythonProjects/kbpedia/v300/build_ins/stubs/kbpedia_rc_stub.owl'\n",
    "    kko_file = 'C:/1-PythonProjects/kbpedia/v300/build_ins/stubs/kko.owl'\n",
    "else:\n",
    "    print('You have entered an inaccurate source parameter for the build.')\n",
    "skos_file = 'http://www.w3.org/2004/02/skos/core' \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(**NOTE**: We later add an 'extract' option to the above to integrate this with our earlier extraction routines.)\n",
    "\n",
    "As I covered in [**CWPK #21**](https://www.mkbergman.com/2353/cwpk-21-some-accumulated-tips/), two tricky areas in this project are related to scope. The first tricky area relates to the internal Python scope of **[LEGB](https://realpython.com/python-scope-legb-rule/)**, which stands for *local &rarr; enclosed &rarr; global &rarr; built-in* and means that objects declared on the right are available to the left, but not left to right for the arrows shown. Care is thus needed about how information gets passed between Python program components. So, yes, a bit of that trickiness is in play with this installment, but the broader issues pertain to the second tricky area.\n",
    "\n",
    "The second area is the interplay of imports, ontologies, and namespaces within [owlready2](http://www.lesfleursdunormal.fr/static/informatique/owlready/index_en.html), plus its own internal 'world' namespace. \n",
    "\n",
    "I have struggled to get these distinctions right, and I'm still not sure I have all of the nuances down or correct. But, here are some things I have learned in *cowpoke*.\n",
    "\n",
    "First, when loading an ontology, I give it a 'world' namespace assigned to the 'World' internal global namespace for owlready2. Since I am only doing *cowpoke*-related development in Python at a given time, I can afford to claim the entire space and perhaps lessen other naming problems. Maybe this is superfluous, but I have found it to be a recipe that works for me.\n",
    "\n",
    "Second, when one imports an ontology into the working ontology (declaring the working ontology being step one), all ontologies available to the import are available to the working ontology. However, if one wants to modify or add items to these imported ontologies, each one needs to be explicity declared, as is done for <code>skos</code> and <code>kko</code> in our current effort. \n",
    "\n",
    "Third, it is essential to declare the namespaces for these imports under the current working ontology. Then, from that point forward, it is also essential to be cognizant that these separate namespaces need to be addressed explicitly. In the case of *cowpoke* and KBpedia, for example, we have classes from our governing upper ontology, KKO (also with namespace '<code>kko</code>') and the reference concepts of the full KBpedia (namespace '<code>rc</code>'). More than one namespace in the working ontology does complicate matters quite a bit, but that is also the more realistic architecture and design approach. Part of the nature of semantic technologies is to promote interoperability among multiple knowledge graphs or ontologies, each of which will have at least one of its own namespaces. To do meaningful work across ontologies, it is important to understand these ontology &larr; &rarr; namespace distinctions.\n",
    "\n",
    "This is how these assignments needed to work out for our build routines based on these considerations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "kb = world.get_ontology(kbpedia).load()\n",
    "rc = kb.get_namespace('http://kbpedia.org/kko/rc/')                # need to make sure we set the namespace\n",
    "\n",
    "skos = world.get_ontology(skos_file).load()\n",
    "kb.imported_ontologies.append(skos)\n",
    "core = world.get_namespace('http://www.w3.org/2004/02/skos/core#')\n",
    "\n",
    "kko = world.get_ontology(kko_file).load()\n",
    "kb.imported_ontologies.append(kko)\n",
    "kko = kb.get_namespace('http://kbpedia.org/ontologies/kko#')       # need to assign namespace to main onto ('kb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have set up our initial build switches and defined our ontologies and related namespaces, we are ready to construct the code for our first build attempt. In this instance, we will be working with only a single class structure input file to the build, <code>typol_AudioInfo.csv</code>, which according to our 'start' build switch (see above) is found in the <code>kbpedia/v300/build_ins/typologies/</code> directory under our project location.\n",
    "\n",
    "The routine below needs to go through three different passes (at least as I have naively specified it!), and is fairly complicated. There are quite a few notes below the code listing explaining some of these steps. Also note we will be definining this code block as a function and the <code>import types</code> statement will be moved to the header in our eventual build module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import types\n",
    "\n",
    "src_file = 'C:/1-PythonProjects/kbpedia/v300/build_ins/typologies/typol_AudioInfo.csv'\n",
    "kko_list = typol_dict.values()\n",
    "with open(src_file, 'r', encoding='utf8') as csv_file:                 # Note 1\n",
    "    is_first_row = True\n",
    "    reader = csv.DictReader(csv_file, delimiter=',', fieldnames=['id', 'subClassOf', 'parent'])                 \n",
    "    for row in reader:                                                 ## Note 2: Pass 1: register class\n",
    "        id = row['id']                                                 # Note 3\n",
    "        parent = row['parent']                                         # Note 3\n",
    "        id = id.replace('http://kbpedia.org/kko/rc/', 'rc.')           # Note 4\n",
    "        id = id.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        id_frag = id.replace('rc.', '')\n",
    "        id_frag = id_frag.replace('kko.', '')\n",
    "        parent = parent.replace('http://kbpedia.org/kko/rc/', 'rc.') \n",
    "        parent = parent.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        parent = parent.replace('owl:', 'owl.')\n",
    "        parent_frag = parent.replace('rc.', '')\n",
    "        parent_frag = parent_frag.replace('kko.', '')\n",
    "        parent_frag = parent_frag.replace('owl.', '')\n",
    "        if is_first_row:                                               # Note 5\n",
    "            is_first_row = False\n",
    "            continue      \n",
    "        with rc:                                                       # Note 6\n",
    "            kko_id = None\n",
    "            kko_frag = None\n",
    "            if parent_frag == 'Thing':                                 # Note 7                               \n",
    "                if id in kko_list:                                     # Note 8\n",
    "                    kko_id = id\n",
    "                    kko_frag = id_frag\n",
    "                else:    \n",
    "                    id = types.new_class(id_frag, (Thing,))            # Note 6\n",
    "        if kko_id != None:                                             # Note 8\n",
    "            with kko:                                                  # same form as Note 6\n",
    "                kko_id = types.new_class(kko_frag, (Thing,))  \n",
    "with open(src_file, 'r', encoding='utf8') as csv_file:\n",
    "    is_first_row = True\n",
    "    reader = csv.DictReader(csv_file, delimiter=',', fieldnames=['id', 'subClassOf', 'parent'])\n",
    "    for row in reader:                                                 ## Note 2: Pass 2: assign parent\n",
    "        id = row['id']\n",
    "        parent = row['parent']\n",
    "        id = id.replace('http://kbpedia.org/kko/rc/', 'rc.')           # Note 4\n",
    "        id = id.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        id_frag = id.replace('rc.', '')\n",
    "        id_frag = id_frag.replace('kko.', '')\n",
    "        parent = parent.replace('http://kbpedia.org/kko/rc/', 'rc.') \n",
    "        parent = parent.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        parent = parent.replace('owl:', 'owl.')\n",
    "        parent_frag = parent.replace('rc.', '')\n",
    "        parent_frag = parent_frag.replace('kko.', '')\n",
    "        parent_frag = parent_frag.replace('owl.', '')\n",
    "        if is_first_row:\n",
    "            is_first_row = False\n",
    "            continue          \n",
    "        with rc:\n",
    "            kko_id = None                                              # Note 9\n",
    "            kko_frag = None\n",
    "            kko_parent = None\n",
    "            kko_parent_frag = None\n",
    "            if parent_frag is not 'Thing':                             # Note 10\n",
    "                if parent in kko_list:\n",
    "                    kko_id = id\n",
    "                    kko_frag = id_frag\n",
    "                    kko_parent = parent\n",
    "                    kko_parent_frag = parent_frag\n",
    "                else:   \n",
    "                    var1 = getattr(rc, id_frag)                        # Note 11\n",
    "                    var2 = getattr(rc, parent_frag)\n",
    "                    if var2 == None:                                   # Note 12\n",
    "                        continue\n",
    "                    else:\n",
    "                        var1.is_a.append(var2)                         # Note 13\n",
    "        if kko_parent != None:                                         # Note 14        \n",
    "            with kko:                \n",
    "                if kko_id in kko_list:                                 # Note 15\n",
    "                    continue\n",
    "                else:\n",
    "                    var1 = getattr(rc, kko_frag)                       # Note 16\n",
    "                    var2 = getattr(kko, kko_parent_frag)\n",
    "                    var1.is_a.append(var2)\n",
    "thing_list = []                                                        # Note 17\n",
    "with open(src_file, 'r', encoding='utf8') as csv_file:\n",
    "    is_first_row = True\n",
    "    reader = csv.DictReader(csv_file, delimiter=',', fieldnames=['id', 'subClassOf', 'parent'])\n",
    "    for row in reader:                                                 ## Note 2: Pass 3: remove owl.Thing\n",
    "        id = row['id']\n",
    "        parent = row['parent']\n",
    "        id = id.replace('http://kbpedia.org/kko/rc/', 'rc.')           # Note 4\n",
    "        id = id.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        id_frag = id.replace('rc.', '')\n",
    "        id_frag = id_frag.replace('kko.', '')\n",
    "        parent = parent.replace('http://kbpedia.org/kko/rc/', 'rc.') \n",
    "        parent = parent.replace('http://kbpedia.org/ontologies/kko#', 'kko.')\n",
    "        parent = parent.replace('owl:', 'owl.')\n",
    "        parent_frag = parent.replace('rc.', '')\n",
    "        parent_frag = parent_frag.replace('kko.', '')\n",
    "        parent_frag = parent_frag.replace('owl.', '')\n",
    "        if is_first_row:\n",
    "            is_first_row = False\n",
    "            continue\n",
    "        if parent_frag == 'Thing':                                     # Note 18\n",
    "            if id in thing_list:                                       # Note 17\n",
    "                continue\n",
    "            else:\n",
    "                if id in kko_list:                                     # Note 19\n",
    "                    var1 = getattr(kko, id_frag)\n",
    "                    thing_list.append(id)\n",
    "                else:                                                  # Note 19\n",
    "                    var1 = getattr(rc, id_frag)\n",
    "                    var1.is_a.remove(owl.Thing)\n",
    "                    thing_list.append(id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code block above was the most challenging to date in this **CWPK** series. Some of the lessons from working this out are offered in [**CWPK #21**](https://www.mkbergman.com/2353/cwpk-21-some-accumulated-tips/). Here are the notes that correspond to some of the statements made in the code above:\n",
    "\n",
    "1. This is a fairly standard CSV processing routine. However, note the 'fieldnames' that are assigned, which give us a basis as the routine proceeds to pick out individual column values by row\n",
    "\n",
    "\n",
    "2. Each file processed requires three passes: *Pass #1* - registers each new item in the source file as a bona fide <code>owl:Class</code>; *Pass #2* - each new item, now properly registered to the system, is assigned its parent class; and *Pass #3* - each of the new items has its direct assignment to <code>owl:Class</code> removed to provide a cleaner hierarchy layout\n",
    "\n",
    "\n",
    "3. We are assigning each row value to a local variable for processing during the loop\n",
    "\n",
    "\n",
    "4. In this, and in the lines to follow, we are reducing the class string and its parent string from potentially its full IRI string to prefix + Name. This gives us the flexibility to have different format input files. We will eventually pull this repeated code each loop out into its own function\n",
    "\n",
    "\n",
    "5. This is a standard approach in CSV file processing to skip the first header row in the file\n",
    "\n",
    "\n",
    "6. There are a few methods apparently possible in owlready2 for assigning a class, but this form of looping over the ontology using the '<code>rc</code>' namespace is the only version I was able to get to work successfully, with the assignment statement as shown in the second part of this method. Note the assignment to 'Thing' is in the form of a tuple, which is why there is a trailing comma\n",
    "\n",
    "\n",
    "7. Via this check, we only pick up the initial class declarations in our input file, and skip over all of the others that set actual direct parents (which we deal with in *Pass #2*)\n",
    "\n",
    "\n",
    "8. We check all of our input roles to see if the row class is already in our kko dictionary (kko_list, set above the routine) or not. If it is a <code>kko.Class</code>, we assign the row information to a new variable, which we then process outside of the 'rc' loop so as to not get the namespaces confused\n",
    "\n",
    "\n",
    "9. Initializing all of this loops variables to 'None'\n",
    "\n",
    "\n",
    "10. Same processing checks as for *Pass #1*, except now we are checking on the parent values\n",
    "\n",
    "\n",
    "11. This is an owlready2 tip, and a critical one, for getting a class type value from a string input; without this, the class assignment method (Note 13) fails\n",
    "\n",
    "\n",
    "12. If var2 is not in the '<code>rc</code>' namespace (in other words, it is in '<code>kko</code>', we skip the parent assignment in the '<code>rc</code>' loop\n",
    "\n",
    "\n",
    "13. This is another owlready2 method for assigning a class to a parent class. In this loop given the checks performed, both parent and id are in the '<code>rc</code>' namespace\n",
    "\n",
    "\n",
    "14. As for *Pass #1*, we are now processing the '<code>kko</code>' namespace items outside of the '<code>rc</code>' namespace and in its own '<code>kko</code>' namespace\n",
    "\n",
    "\n",
    "15. We earlier picked up rows with parents in the '<code>kko</code>' namespace; via this call, we also exclude rows with a '<code>kko</code>' id as well, since our imported KKO ontology already has all kko class assignments set\n",
    "\n",
    "\n",
    "16. We use the same parent class assignment method as in Note #11, but now for ids in the '<code>rc</code>' namespace and parents in the '<code>kko</code>' namespace. However, the routine so far also results in a long listing of classes directly under <code>owl:Thing</code> root **(1)** in an ontology editor such as [Protégé](https://en.wikipedia.org/wiki/Prot%C3%A9g%C3%A9_(software)):\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/structural-ingest-1.png\" title=\"Class Import with Duplicate owl:Thing Assignments\" width=\"800\" alt=\"Class Import with Duplicate owl:Thing Assignments\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 1: Class Import with Duplicate owl:Thing Assignments\n",
    "\n",
    "</div>\n",
    "\n",
    "\n",
    "17. We use a 'thing_list', and assign it as an empty set at the beginning of the *Pass #3* routine, because we will be deleting class assignments to <code>owl:Thing</code>. There may be multiple declarations in our build file, but we only may delete the assignment once from the knowledge base. The lookup to 'thing_list' prevents us from erroring when trying to delete for a second or more times \n",
    "\n",
    "\n",
    "18. We are selecting on 'Thing' because we want to *unassign* all of the temporary <code>owl:Thing</code> class assignments needed to provide placeholders in *Pass #1* (**Note**: recall in our structure extractor routines in [**CWPK #28**](https://www.mkbergman.com/2363/cwpk-28-extracting-structure-for-typologies/) we added an extra assignment to add an <code>owl:Thing</code> class definition so that all classes in the extracted files could be recognized and loaded by external ontology editors)\n",
    "\n",
    "19. We differentiate between '<code>rc</code>' and '<code>kko</code>' concepts because the kko are defined separated in the KKO ontology, used as one of our build stubs.\n",
    "\n",
    "\n",
    "As you run this routine in real time from Jupyter Notebook, you can inspect what have been removed by inspecting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(thing_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now inspect this loading of an individual typology into our stub. We need to preface our 'save' statement with the 'kb' ontology identifier. I also have chosen to use the 'working' directory for saving these temporary results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "kb.save(file=r'C:/1-PythonProjects/kbpedia/v300/build_ins/working/kbpedia_reference_concepts.owl', format=\"rdfxml\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, phew! After much time and trial, I was able to get this code running successfully! Here is the output of the full routine:\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/structural-ingest-2.png\" title=\"Class Import with Proper Hierarchical Placement\" width=\"800\" alt=\"Class Import with Proper Hierarchical Placement\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 2: Class Import with Proper Hierarchical Placement\n",
    "\n",
    "</div>\n",
    "\n",
    "We can see that our flat listing under the root is now gone **(1)** and all concepts are properly organized according to the proper structure in the KKO hierarchy **(2)**.\n",
    "\n",
    "We now have a template for looping over multiple typologies to contribute to a build as well as to bring in KBpedia's property structure. These are the topics of our next installment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "There were dozens of small issues and problems that arose in working out the routine above. Here are some resources that were especially helpful in informing that effort:\n",
    "\n",
    "- The [owlready2 mailing list](http://owlready.8326.n8.nabble.com/) is an increasingly helpful resource for specific questions. With regard to the above, searches on 'namespace' and 'import' were helpful\n",
    "- [CSV file Import with DictReader](https://apptension.com/blog/2018/05/07/import-csv-file-in-python/)\n",
    "- [Traps for the Unwary in Python’s Import System](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html)\n",
    "- DataCamp's [Scope of Variables in Python](https://www.datacamp.com/community/tutorials/scope-of-variables-python)\n",
    "- Python's [CSV File Reading and Writing](https://docs.python.org/3/library/csv.html) library.\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}