use-case-microscopy-scans.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Case: Microscopy Scans Database\n",
    "This notebook demonstates a usage example of the CHiSEL schema evolution framework. In this example, a simple single-table database of microscopy scans is evolved into a normalized database with attributes aligned to newly created vocabulary term sets.\n",
    "\n",
    "The example database used here was created by denormalizing the database reported in earlier work [1]. This effectively returned this database to its prior state in early development before it evolved through iterative changes over the lifecycle of the project. We then demonstrate the evolution of the database as it could have been performed with the chisel library.\n",
    "\n",
    "1. Schuler, R E and Kesselman, C and Czajkowski, K. 2014. Digital asset management for heterogeneous biomedical data in an era of data-intensive science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import chisel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to the denormalized scans database. (Authentication required)\n",
    "catalog = chisel.connect('https://cirm-dev.isrd.isi.edu/ermrest/catalog/4')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### List of Tables\n",
       "| Schema | Name           | Kind  | Comment |\n",
       "|--------|----------------|-------|---------|\n",
       "| public | ERMrest\\_Client | table | None    |\n",
       "| public | ERMrest\\_Group | table | None    |\n",
       "| public | Scans          | table | None    |\n"
      ],
      "text/plain": [
       "### List of Tables\n",
       "| Schema | Name           | Kind  | Comment |\n",
       "|--------|----------------|-------|---------|\n",
       "| public | ERMrest_Client | table | None    |\n",
       "| public | ERMrest_Group  | table | None    |\n",
       "| public | Scans          | table | None    |"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "catalog['public'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get handle to the 'Scans' table.\n",
    "Scans = catalog['public']['Scans']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"public.Scans\"\n",
       "| Column                       | Type        | Nullable | Default                                                       | Comment |\n",
       "|------------------------------|-------------|----------|---------------------------------------------------------------|---------|\n",
       "| RID                          | ermrest\\_rid | False    | \\_ermrest\\.urlb32\\_encode\\(nextval\\('\\_ermrest\\.rid\\_seq'::regclass\\)\\) | None    |\n",
       "| RCT                          | ermrest\\_rct | False    | now\\(\\)                                                       | None    |\n",
       "| RMT                          | ermrest\\_rmt | False    | now\\(\\)                                                       | None    |\n",
       "| RCB                          | ermrest\\_rcb | True     | \\_ermrest\\.current\\_client\\(\\)                                | None    |\n",
       "| RMB                          | ermrest\\_rmb | True     | \\_ermrest\\.current\\_client\\(\\)                                | None    |\n",
       "| Scan:slide\\_id               | text        | True     | None                                                          | None    |\n",
       "| Scan:filename                | text        | True     | None                                                          | None    |\n",
       "| Scan:HTTP URL                | text        | True     | None                                                          | None    |\n",
       "| Scan:bytes                   | text        | True     | None                                                          | None    |\n",
       "| Scan:Thumbnail               | text        | True     | None                                                          | None    |\n",
       "| Scan:Microscope              | text        | True     | None                                                          | None    |\n",
       "| Scan:Camera                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Objective               | text        | True     | None                                                          | None    |\n",
       "| Scan:Exposure Time           | text        | True     | None                                                          | None    |\n",
       "| Scan:Channels                | text        | True     | None                                                          | None    |\n",
       "| Scan:Channel Name            | text        | True     | None                                                          | None    |\n",
       "| Scan:Contrast Method         | text        | True     | None                                                          | None    |\n",
       "| Scan:Magnification           | text        | True     | None                                                          | None    |\n",
       "| Scan:Light Source Intensity  | text        | True     | None                                                          | None    |\n",
       "| Scan:Image Size              | text        | True     | None                                                          | None    |\n",
       "| Scan:Scaling \\(per pixel\\)   | text        | True     | None                                                          | None    |\n",
       "| Scan:DZI                     | text        | True     | None                                                          | None    |\n",
       "| Scan:czi2dzi                 | text        | True     | None                                                          | None    |\n",
       "| Scan:File Date               | text        | True     | None                                                          | None    |\n",
       "| Scan:Acquisition Date        | text        | True     | None                                                          | None    |\n",
       "| Scan:id                      | text        | True     | None                                                          | None    |\n",
       "| Scan:species                 | text        | True     | None                                                          | None    |\n",
       "| Scan:age                     | text        | True     | None                                                          | None    |\n",
       "| Scan:gene                    | text        | True     | None                                                          | None    |\n",
       "| Scan:checksum                | text        | True     | None                                                          | None    |\n",
       "| Scan:probe                   | text        | True     | None                                                          | None    |\n",
       "| Scan:accession\\_number       | text        | True     | None                                                          | None    |\n",
       "| Scan:doi                     | text        | True     | None                                                          | None    |\n",
       "| Scan:ark                     | text        | True     | None                                                          | None    |\n",
       "| Scan:mime\\_type              | text        | True     | None                                                          | None    |\n",
       "| Scan:description             | text        | True     | None                                                          | None    |\n",
       "| Scan:submitter               | text        | True     | None                                                          | None    |\n",
       "| Scan:submitted               | text        | True     | None                                                          | None    |\n",
       "| Scan:public\\_release\\_date   | text        | True     | None                                                          | None    |\n",
       "| Scan:tissue                  | text        | True     | None                                                          | None    |\n",
       "| Scan:gender                  | text        | True     | None                                                          | None    |\n",
       "| Scan:age\\_stage              | text        | True     | None                                                          | None    |\n",
       "| Scan:specimen\\_fixation      | text        | True     | None                                                          | None    |\n",
       "| Scan:embedding\\_medium       | text        | True     | None                                                          | None    |\n",
       "| Scan:staining\\_protocol      | text        | True     | None                                                          | None    |\n",
       "| Scan:resolution              | text        | True     | None                                                          | None    |\n",
       "| Scan:uri                     | text        | True     | None                                                          | None    |\n",
       "| Scan:status                  | text        | True     | None                                                          | None    |\n",
       "| Scan:last\\_modified          | text        | True     | None                                                          | None    |\n",
       "| Scan:age\\_rank               | text        | True     | None                                                          | None    |\n",
       "| Scan:Disambiguator           | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment ID           | text        | True     | None                                                          | None    |\n",
       "| Scan:Specimen ID             | text        | True     | None                                                          | None    |\n",
       "| Scan:Probes                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment Type         | text        | True     | None                                                          | None    |\n",
       "| Slide:ID                     | text        | True     | None                                                          | None    |\n",
       "| Slide:Seq\\.                  | text        | True     | None                                                          | None    |\n",
       "| Slide:Specimen ID            | text        | True     | None                                                          | None    |\n",
       "| Slide:Experiment ID          | text        | True     | None                                                          | None    |\n",
       "| Slide:Number of Scans        | text        | True     | None                                                          | None    |\n",
       "| Slide:Label                  | text        | True     | None                                                          | None    |\n",
       "| Specimen:ID                  | text        | True     | None                                                          | None    |\n",
       "| Specimen:Section Date        | text        | True     | None                                                          | None    |\n",
       "| Specimen:Initials            | text        | True     | None                                                          | None    |\n",
       "| Specimen:Disambiguator       | text        | True     | None                                                          | None    |\n",
       "| Specimen:Species             | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age                 | text        | True     | None                                                          | None    |\n",
       "| Specimen:Tissue              | text        | True     | None                                                          | None    |\n",
       "| Specimen:Gene                | text        | True     | None                                                          | None    |\n",
       "| Specimen:Specimen Identifier | text        | True     | None                                                          | None    |\n",
       "| Specimen:Genes               | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age Unit            | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age Value           | text        | True     | None                                                          | None    |\n",
       "| Specimen:age\\_rank           | text        | True     | None                                                          | None    |\n",
       "| Specimen:Number of Slides    | text        | True     | None                                                          | None    |\n",
       "| Specimen:Number of Scans     | text        | True     | None                                                          | None    |\n",
       "| Specimen:Label               | text        | True     | None                                                          | None    |\n",
       "| Experiment:ID                | text        | True     | None                                                          | None    |\n",
       "| Experiment:Experiment Date   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Initials          | text        | True     | None                                                          | None    |\n",
       "| Experiment:Disambiguator     | text        | True     | None                                                          | None    |\n",
       "| Experiment:Experiment Type   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Probe             | text        | True     | None                                                          | None    |\n",
       "| Experiment:Number of Slides  | text        | True     | None                                                          | None    |\n",
       "| Experiment:Number of Scans   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Probes            | text        | True     | None                                                          | None    |\n"
      ],
      "text/plain": [
       "### Table \"public.Scans\"\n",
       "| Column                       | Type        | Nullable | Default                                                       | Comment |\n",
       "|------------------------------|-------------|----------|---------------------------------------------------------------|---------|\n",
       "| RID                          | ermrest_rid | False    | _ermrest.urlb32_encode(nextval('_ermrest.rid_seq'::regclass)) | None    |\n",
       "| RCT                          | ermrest_rct | False    | now()                                                         | None    |\n",
       "| RMT                          | ermrest_rmt | False    | now()                                                         | None    |\n",
       "| RCB                          | ermrest_rcb | True     | _ermrest.current_client()                                     | None    |\n",
       "| RMB                          | ermrest_rmb | True     | _ermrest.current_client()                                     | None    |\n",
       "| Scan:slide_id                | text        | True     | None                                                          | None    |\n",
       "| Scan:filename                | text        | True     | None                                                          | None    |\n",
       "| Scan:HTTP URL                | text        | True     | None                                                          | None    |\n",
       "| Scan:bytes                   | text        | True     | None                                                          | None    |\n",
       "| Scan:Thumbnail               | text        | True     | None                                                          | None    |\n",
       "| Scan:Microscope              | text        | True     | None                                                          | None    |\n",
       "| Scan:Camera                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Objective               | text        | True     | None                                                          | None    |\n",
       "| Scan:Exposure Time           | text        | True     | None                                                          | None    |\n",
       "| Scan:Channels                | text        | True     | None                                                          | None    |\n",
       "| Scan:Channel Name            | text        | True     | None                                                          | None    |\n",
       "| Scan:Contrast Method         | text        | True     | None                                                          | None    |\n",
       "| Scan:Magnification           | text        | True     | None                                                          | None    |\n",
       "| Scan:Light Source Intensity  | text        | True     | None                                                          | None    |\n",
       "| Scan:Image Size              | text        | True     | None                                                          | None    |\n",
       "| Scan:Scaling (per pixel)     | text        | True     | None                                                          | None    |\n",
       "| Scan:DZI                     | text        | True     | None                                                          | None    |\n",
       "| Scan:czi2dzi                 | text        | True     | None                                                          | None    |\n",
       "| Scan:File Date               | text        | True     | None                                                          | None    |\n",
       "| Scan:Acquisition Date        | text        | True     | None                                                          | None    |\n",
       "| Scan:id                      | text        | True     | None                                                          | None    |\n",
       "| Scan:species                 | text        | True     | None                                                          | None    |\n",
       "| Scan:age                     | text        | True     | None                                                          | None    |\n",
       "| Scan:gene                    | text        | True     | None                                                          | None    |\n",
       "| Scan:checksum                | text        | True     | None                                                          | None    |\n",
       "| Scan:probe                   | text        | True     | None                                                          | None    |\n",
       "| Scan:accession_number        | text        | True     | None                                                          | None    |\n",
       "| Scan:doi                     | text        | True     | None                                                          | None    |\n",
       "| Scan:ark                     | text        | True     | None                                                          | None    |\n",
       "| Scan:mime_type               | text        | True     | None                                                          | None    |\n",
       "| Scan:description             | text        | True     | None                                                          | None    |\n",
       "| Scan:submitter               | text        | True     | None                                                          | None    |\n",
       "| Scan:submitted               | text        | True     | None                                                          | None    |\n",
       "| Scan:public_release_date     | text        | True     | None                                                          | None    |\n",
       "| Scan:tissue                  | text        | True     | None                                                          | None    |\n",
       "| Scan:gender                  | text        | True     | None                                                          | None    |\n",
       "| Scan:age_stage               | text        | True     | None                                                          | None    |\n",
       "| Scan:specimen_fixation       | text        | True     | None                                                          | None    |\n",
       "| Scan:embedding_medium        | text        | True     | None                                                          | None    |\n",
       "| Scan:staining_protocol       | text        | True     | None                                                          | None    |\n",
       "| Scan:resolution              | text        | True     | None                                                          | None    |\n",
       "| Scan:uri                     | text        | True     | None                                                          | None    |\n",
       "| Scan:status                  | text        | True     | None                                                          | None    |\n",
       "| Scan:last_modified           | text        | True     | None                                                          | None    |\n",
       "| Scan:age_rank                | text        | True     | None                                                          | None    |\n",
       "| Scan:Disambiguator           | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment ID           | text        | True     | None                                                          | None    |\n",
       "| Scan:Specimen ID             | text        | True     | None                                                          | None    |\n",
       "| Scan:Probes                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment Type         | text        | True     | None                                                          | None    |\n",
       "| Slide:ID                     | text        | True     | None                                                          | None    |\n",
       "| Slide:Seq.                   | text        | True     | None                                                          | None    |\n",
       "| Slide:Specimen ID            | text        | True     | None                                                          | None    |\n",
       "| Slide:Experiment ID          | text        | True     | None                                                          | None    |\n",
       "| Slide:Number of Scans        | text        | True     | None                                                          | None    |\n",
       "| Slide:Label                  | text        | True     | None                                                          | None    |\n",
       "| Specimen:ID                  | text        | True     | None                                                          | None    |\n",
       "| Specimen:Section Date        | text        | True     | None                                                          | None    |\n",
       "| Specimen:Initials            | text        | True     | None                                                          | None    |\n",
       "| Specimen:Disambiguator       | text        | True     | None                                                          | None    |\n",
       "| Specimen:Species             | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age                 | text        | True     | None                                                          | None    |\n",
       "| Specimen:Tissue              | text        | True     | None                                                          | None    |\n",
       "| Specimen:Gene                | text        | True     | None                                                          | None    |\n",
       "| Specimen:Specimen Identifier | text        | True     | None                                                          | None    |\n",
       "| Specimen:Genes               | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age Unit            | text        | True     | None                                                          | None    |\n",
       "| Specimen:Age Value           | text        | True     | None                                                          | None    |\n",
       "| Specimen:age_rank            | text        | True     | None                                                          | None    |\n",
       "| Specimen:Number of Slides    | text        | True     | None                                                          | None    |\n",
       "| Specimen:Number of Scans     | text        | True     | None                                                          | None    |\n",
       "| Specimen:Label               | text        | True     | None                                                          | None    |\n",
       "| Experiment:ID                | text        | True     | None                                                          | None    |\n",
       "| Experiment:Experiment Date   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Initials          | text        | True     | None                                                          | None    |\n",
       "| Experiment:Disambiguator     | text        | True     | None                                                          | None    |\n",
       "| Experiment:Experiment Type   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Probe             | text        | True     | None                                                          | None    |\n",
       "| Experiment:Number of Slides  | text        | True     | None                                                          | None    |\n",
       "| Experiment:Number of Scans   | text        | True     | None                                                          | None    |\n",
       "| Experiment:Probes            | text        | True     | None                                                          | None    |"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at the current denormalized state.\n",
    "Scans.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Has this many columns: 86\n"
     ]
    }
   ],
   "source": [
    "print('Has this many columns:', len(Scans.columns.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans\"\n",
       "| Column                | Type | Nullable | Default | Comment |\n",
       "|-----------------------|------|----------|---------|---------|\n",
       "| Slide:ID              | text | True     | None    | None    |\n",
       "| Slide:Seq\\.           | text | True     | None    | None    |\n",
       "| Slide:Specimen ID     | text | True     | None    | None    |\n",
       "| Slide:Experiment ID   | text | True     | None    | None    |\n",
       "| Slide:Number of Scans | text | True     | None    | None    |\n",
       "| Slide:Label           | text | True     | None    | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans\"\n",
       "| Column                | Type | Nullable | Default | Comment |\n",
       "|-----------------------|------|----------|---------|---------|\n",
       "| Slide:ID              | text | True     | None    | None    |\n",
       "| Slide:Seq.            | text | True     | None    | None    |\n",
       "| Slide:Specimen ID     | text | True     | None    | None    |\n",
       "| Slide:Experiment ID   | text | True     | None    | None    |\n",
       "| Slide:Number of Scans | text | True     | None    | None    |\n",
       "| Slide:Label           | text | True     | None    | None    |"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 1: Reify the Slides\n",
    "Slides = Scans.reify(\n",
    "  {Scans['Slide:ID']},\n",
    "  {column for column in Scans.columns.values() \n",
    "   if column.name.startswith('Slide:') and column.name != 'Slide:ID'\n",
    "  }\n",
    ")\n",
    "Slides.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans\"\n",
       "| Column                       | Type | Nullable | Default | Comment |\n",
       "|------------------------------|------|----------|---------|---------|\n",
       "| Specimen:ID                  | text | True     | None    | None    |\n",
       "| Specimen:Section Date        | text | True     | None    | None    |\n",
       "| Specimen:Initials            | text | True     | None    | None    |\n",
       "| Specimen:Disambiguator       | text | True     | None    | None    |\n",
       "| Specimen:Species             | text | True     | None    | None    |\n",
       "| Specimen:Age                 | text | True     | None    | None    |\n",
       "| Specimen:Tissue              | text | True     | None    | None    |\n",
       "| Specimen:Gene                | text | True     | None    | None    |\n",
       "| Specimen:Specimen Identifier | text | True     | None    | None    |\n",
       "| Specimen:Genes               | text | True     | None    | None    |\n",
       "| Specimen:Age Unit            | text | True     | None    | None    |\n",
       "| Specimen:Age Value           | text | True     | None    | None    |\n",
       "| Specimen:age\\_rank           | text | True     | None    | None    |\n",
       "| Specimen:Number of Slides    | text | True     | None    | None    |\n",
       "| Specimen:Number of Scans     | text | True     | None    | None    |\n",
       "| Specimen:Label               | text | True     | None    | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans\"\n",
       "| Column                       | Type | Nullable | Default | Comment |\n",
       "|------------------------------|------|----------|---------|---------|\n",
       "| Specimen:ID                  | text | True     | None    | None    |\n",
       "| Specimen:Section Date        | text | True     | None    | None    |\n",
       "| Specimen:Initials            | text | True     | None    | None    |\n",
       "| Specimen:Disambiguator       | text | True     | None    | None    |\n",
       "| Specimen:Species             | text | True     | None    | None    |\n",
       "| Specimen:Age                 | text | True     | None    | None    |\n",
       "| Specimen:Tissue              | text | True     | None    | None    |\n",
       "| Specimen:Gene                | text | True     | None    | None    |\n",
       "| Specimen:Specimen Identifier | text | True     | None    | None    |\n",
       "| Specimen:Genes               | text | True     | None    | None    |\n",
       "| Specimen:Age Unit            | text | True     | None    | None    |\n",
       "| Specimen:Age Value           | text | True     | None    | None    |\n",
       "| Specimen:age_rank            | text | True     | None    | None    |\n",
       "| Specimen:Number of Slides    | text | True     | None    | None    |\n",
       "| Specimen:Number of Scans     | text | True     | None    | None    |\n",
       "| Specimen:Label               | text | True     | None    | None    |"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 2: Reify the Specimens\n",
    "Specimens = Scans.reify(\n",
    "  {Scans['Specimen:ID']},\n",
    "  {column for column in Scans.columns.values() \n",
    "   if column.name.startswith('Specimen:') and column.name != 'Specimen:ID'\n",
    "  }\n",
    ")\n",
    "Specimens.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans\"\n",
       "| Column                      | Type | Nullable | Default | Comment |\n",
       "|-----------------------------|------|----------|---------|---------|\n",
       "| Experiment:ID               | text | True     | None    | None    |\n",
       "| Experiment:Experiment Date  | text | True     | None    | None    |\n",
       "| Experiment:Initials         | text | True     | None    | None    |\n",
       "| Experiment:Disambiguator    | text | True     | None    | None    |\n",
       "| Experiment:Experiment Type  | text | True     | None    | None    |\n",
       "| Experiment:Probe            | text | True     | None    | None    |\n",
       "| Experiment:Number of Slides | text | True     | None    | None    |\n",
       "| Experiment:Number of Scans  | text | True     | None    | None    |\n",
       "| Experiment:Probes           | text | True     | None    | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans\"\n",
       "| Column                      | Type | Nullable | Default | Comment |\n",
       "|-----------------------------|------|----------|---------|---------|\n",
       "| Experiment:ID               | text | True     | None    | None    |\n",
       "| Experiment:Experiment Date  | text | True     | None    | None    |\n",
       "| Experiment:Initials         | text | True     | None    | None    |\n",
       "| Experiment:Disambiguator    | text | True     | None    | None    |\n",
       "| Experiment:Experiment Type  | text | True     | None    | None    |\n",
       "| Experiment:Probe            | text | True     | None    | None    |\n",
       "| Experiment:Number of Slides | text | True     | None    | None    |\n",
       "| Experiment:Number of Scans  | text | True     | None    | None    |\n",
       "| Experiment:Probes           | text | True     | None    | None    |"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 3: Reify the Experiments\n",
    "Experiments = Scans.reify(\n",
    "  {Scans['Experiment:ID']},\n",
    "  {column for column in Scans.columns.values() \n",
    "   if column.name.startswith('Experiment:') and column.name != 'Experiment:ID'\n",
    "  }\n",
    ")\n",
    "Experiments.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans\"\n",
       "| Column                      | Type        | Nullable | Default                                                       | Comment |\n",
       "|-----------------------------|-------------|----------|---------------------------------------------------------------|---------|\n",
       "| RID                         | ermrest\\_rid | False    | \\_ermrest\\.urlb32\\_encode\\(nextval\\('\\_ermrest\\.rid\\_seq'::regclass\\)\\) | None    |\n",
       "| RCT                         | ermrest\\_rct | False    | now\\(\\)                                                       | None    |\n",
       "| RMT                         | ermrest\\_rmt | False    | now\\(\\)                                                       | None    |\n",
       "| RCB                         | ermrest\\_rcb | True     | \\_ermrest\\.current\\_client\\(\\)                                | None    |\n",
       "| RMB                         | ermrest\\_rmb | True     | \\_ermrest\\.current\\_client\\(\\)                                | None    |\n",
       "| Scan:slide\\_id              | text        | True     | None                                                          | None    |\n",
       "| Scan:filename               | text        | True     | None                                                          | None    |\n",
       "| Scan:HTTP URL               | text        | True     | None                                                          | None    |\n",
       "| Scan:bytes                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Thumbnail              | text        | True     | None                                                          | None    |\n",
       "| Scan:Microscope             | text        | True     | None                                                          | None    |\n",
       "| Scan:Camera                 | text        | True     | None                                                          | None    |\n",
       "| Scan:Objective              | text        | True     | None                                                          | None    |\n",
       "| Scan:Exposure Time          | text        | True     | None                                                          | None    |\n",
       "| Scan:Channels               | text        | True     | None                                                          | None    |\n",
       "| Scan:Channel Name           | text        | True     | None                                                          | None    |\n",
       "| Scan:Contrast Method        | text        | True     | None                                                          | None    |\n",
       "| Scan:Magnification          | text        | True     | None                                                          | None    |\n",
       "| Scan:Light Source Intensity | text        | True     | None                                                          | None    |\n",
       "| Scan:Image Size             | text        | True     | None                                                          | None    |\n",
       "| Scan:Scaling \\(per pixel\\)  | text        | True     | None                                                          | None    |\n",
       "| Scan:DZI                    | text        | True     | None                                                          | None    |\n",
       "| Scan:czi2dzi                | text        | True     | None                                                          | None    |\n",
       "| Scan:File Date              | text        | True     | None                                                          | None    |\n",
       "| Scan:Acquisition Date       | text        | True     | None                                                          | None    |\n",
       "| Scan:id                     | text        | True     | None                                                          | None    |\n",
       "| Scan:species                | text        | True     | None                                                          | None    |\n",
       "| Scan:age                    | text        | True     | None                                                          | None    |\n",
       "| Scan:gene                   | text        | True     | None                                                          | None    |\n",
       "| Scan:checksum               | text        | True     | None                                                          | None    |\n",
       "| Scan:probe                  | text        | True     | None                                                          | None    |\n",
       "| Scan:accession\\_number      | text        | True     | None                                                          | None    |\n",
       "| Scan:doi                    | text        | True     | None                                                          | None    |\n",
       "| Scan:ark                    | text        | True     | None                                                          | None    |\n",
       "| Scan:mime\\_type             | text        | True     | None                                                          | None    |\n",
       "| Scan:description            | text        | True     | None                                                          | None    |\n",
       "| Scan:submitter              | text        | True     | None                                                          | None    |\n",
       "| Scan:submitted              | text        | True     | None                                                          | None    |\n",
       "| Scan:public\\_release\\_date  | text        | True     | None                                                          | None    |\n",
       "| Scan:tissue                 | text        | True     | None                                                          | None    |\n",
       "| Scan:gender                 | text        | True     | None                                                          | None    |\n",
       "| Scan:age\\_stage             | text        | True     | None                                                          | None    |\n",
       "| Scan:specimen\\_fixation     | text        | True     | None                                                          | None    |\n",
       "| Scan:embedding\\_medium      | text        | True     | None                                                          | None    |\n",
       "| Scan:staining\\_protocol     | text        | True     | None                                                          | None    |\n",
       "| Scan:resolution             | text        | True     | None                                                          | None    |\n",
       "| Scan:uri                    | text        | True     | None                                                          | None    |\n",
       "| Scan:status                 | text        | True     | None                                                          | None    |\n",
       "| Scan:last\\_modified         | text        | True     | None                                                          | None    |\n",
       "| Scan:age\\_rank              | text        | True     | None                                                          | None    |\n",
       "| Scan:Disambiguator          | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment ID          | text        | True     | None                                                          | None    |\n",
       "| Scan:Specimen ID            | text        | True     | None                                                          | None    |\n",
       "| Scan:Probes                 | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment Type        | text        | True     | None                                                          | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans\"\n",
       "| Column                      | Type        | Nullable | Default                                                       | Comment |\n",
       "|-----------------------------|-------------|----------|---------------------------------------------------------------|---------|\n",
       "| RID                         | ermrest_rid | False    | _ermrest.urlb32_encode(nextval('_ermrest.rid_seq'::regclass)) | None    |\n",
       "| RCT                         | ermrest_rct | False    | now()                                                         | None    |\n",
       "| RMT                         | ermrest_rmt | False    | now()                                                         | None    |\n",
       "| RCB                         | ermrest_rcb | True     | _ermrest.current_client()                                     | None    |\n",
       "| RMB                         | ermrest_rmb | True     | _ermrest.current_client()                                     | None    |\n",
       "| Scan:slide_id               | text        | True     | None                                                          | None    |\n",
       "| Scan:filename               | text        | True     | None                                                          | None    |\n",
       "| Scan:HTTP URL               | text        | True     | None                                                          | None    |\n",
       "| Scan:bytes                  | text        | True     | None                                                          | None    |\n",
       "| Scan:Thumbnail              | text        | True     | None                                                          | None    |\n",
       "| Scan:Microscope             | text        | True     | None                                                          | None    |\n",
       "| Scan:Camera                 | text        | True     | None                                                          | None    |\n",
       "| Scan:Objective              | text        | True     | None                                                          | None    |\n",
       "| Scan:Exposure Time          | text        | True     | None                                                          | None    |\n",
       "| Scan:Channels               | text        | True     | None                                                          | None    |\n",
       "| Scan:Channel Name           | text        | True     | None                                                          | None    |\n",
       "| Scan:Contrast Method        | text        | True     | None                                                          | None    |\n",
       "| Scan:Magnification          | text        | True     | None                                                          | None    |\n",
       "| Scan:Light Source Intensity | text        | True     | None                                                          | None    |\n",
       "| Scan:Image Size             | text        | True     | None                                                          | None    |\n",
       "| Scan:Scaling (per pixel)    | text        | True     | None                                                          | None    |\n",
       "| Scan:DZI                    | text        | True     | None                                                          | None    |\n",
       "| Scan:czi2dzi                | text        | True     | None                                                          | None    |\n",
       "| Scan:File Date              | text        | True     | None                                                          | None    |\n",
       "| Scan:Acquisition Date       | text        | True     | None                                                          | None    |\n",
       "| Scan:id                     | text        | True     | None                                                          | None    |\n",
       "| Scan:species                | text        | True     | None                                                          | None    |\n",
       "| Scan:age                    | text        | True     | None                                                          | None    |\n",
       "| Scan:gene                   | text        | True     | None                                                          | None    |\n",
       "| Scan:checksum               | text        | True     | None                                                          | None    |\n",
       "| Scan:probe                  | text        | True     | None                                                          | None    |\n",
       "| Scan:accession_number       | text        | True     | None                                                          | None    |\n",
       "| Scan:doi                    | text        | True     | None                                                          | None    |\n",
       "| Scan:ark                    | text        | True     | None                                                          | None    |\n",
       "| Scan:mime_type              | text        | True     | None                                                          | None    |\n",
       "| Scan:description            | text        | True     | None                                                          | None    |\n",
       "| Scan:submitter              | text        | True     | None                                                          | None    |\n",
       "| Scan:submitted              | text        | True     | None                                                          | None    |\n",
       "| Scan:public_release_date    | text        | True     | None                                                          | None    |\n",
       "| Scan:tissue                 | text        | True     | None                                                          | None    |\n",
       "| Scan:gender                 | text        | True     | None                                                          | None    |\n",
       "| Scan:age_stage              | text        | True     | None                                                          | None    |\n",
       "| Scan:specimen_fixation      | text        | True     | None                                                          | None    |\n",
       "| Scan:embedding_medium       | text        | True     | None                                                          | None    |\n",
       "| Scan:staining_protocol      | text        | True     | None                                                          | None    |\n",
       "| Scan:resolution             | text        | True     | None                                                          | None    |\n",
       "| Scan:uri                    | text        | True     | None                                                          | None    |\n",
       "| Scan:status                 | text        | True     | None                                                          | None    |\n",
       "| Scan:last_modified          | text        | True     | None                                                          | None    |\n",
       "| Scan:age_rank               | text        | True     | None                                                          | None    |\n",
       "| Scan:Disambiguator          | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment ID          | text        | True     | None                                                          | None    |\n",
       "| Scan:Specimen ID            | text        | True     | None                                                          | None    |\n",
       "| Scan:Probes                 | text        | True     | None                                                          | None    |\n",
       "| Scan:Experiment Type        | text        | True     | None                                                          | None    |"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 4: alter the Scans by dropping the Slides-Speciments-Experiments columns\n",
    "RevisedScans = Scans.select(\n",
    "    *[column for column in Scans.columns.values() \n",
    "      if column.name.startswith('Scan:') or \n",
    "      column.name in ('RID', 'RCB', 'RMB', 'RCT', 'RMT')]\n",
    ")\n",
    "RevisedScans.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scans: 8191\n",
      "Slides: 7814\n",
      "Specimens: 1594\n",
      "Experiments: 1852\n"
     ]
    }
   ],
   "source": [
    "# For demonstration purposes, we can preview the tuples. Here, \n",
    "# we will just print the lengths. Notice that each reified \n",
    "# concept differs in size because they had been repeated in the\n",
    "# denormalized scans table.\n",
    "print(\"Scans:\", len(RevisedScans.fetch()))\n",
    "print(\"Slides:\", len(Slides.fetch()))\n",
    "print(\"Specimens:\", len(Specimens.fetch()))\n",
    "print(\"Experiments:\", len(Experiments.fetch()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans:2ef24ea08d4511e9acd9ac87a3187979\"\n",
       "| Column | Type | Nullable | Default | Comment |\n",
       "|--------|------|----------|---------|---------|\n",
       "| name   | text | True     | None    | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans:2ef24ea08d4511e9acd9ac87a3187979\"\n",
       "| Column | Type | Nullable | Default | Comment |\n",
       "|--------|------|----------|---------|---------|\n",
       "| name   | text | True     | None    | None    |"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 5: Create a custom domain based on the Experiment Type\n",
    "Experiment_Type = Experiments['Experiment:Experiment Type'].to_domain()\n",
    "Experiment_Type.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'SISH'},\n",
       " {'name': 'Other'},\n",
       " {'name': 'H&E'},\n",
       " {'name': 'LACZ'},\n",
       " {'name': 'IF'},\n",
       " {'name': 'TRI'},\n",
       " {'name': 'IHC'},\n",
       " {'name': 'PAS'},\n",
       " {'name': 'RNAscope'},\n",
       " {'name': 'JonesSilver'},\n",
       " {'name': 'V&AB'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview the generated domain.\n",
    "Experiment_Type.fetch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Table \"None.Scans:2efd4b7a8d4511e9acd9ac87a3187979\"\n",
       "| Column   | Type   | Nullable | Default | Comment |\n",
       "|----------|--------|----------|---------|---------|\n",
       "| name     | text   | True     | None    | None    |\n",
       "| synonyms | text\\[\\] | True     | None    | None    |\n"
      ],
      "text/plain": [
       "### Table \"None.Scans:2efd4b7a8d4511e9acd9ac87a3187979\"\n",
       "| Column   | Type   | Nullable | Default | Comment |\n",
       "|----------|--------|----------|---------|---------|\n",
       "| name     | text   | True     | None    | None    |\n",
       "| synonyms | text[] | True     | None    | None    |"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 6: Create a custom vocabulary based on the Specimen Tissue\n",
    "Tissue = Specimens['Specimen:Tissue'].to_vocabulary()\n",
    "Tissue.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Kidney', 'synonyms': ['Kidney']},\n",
       " {'name': 'Neural Tube', 'synonyms': ['Neural Tube']},\n",
       " {'name': 'Brain', 'synonyms': ['Brain']},\n",
       " {'name': 'Other', 'synonyms': ['Other']},\n",
       " {'name': 'Lung', 'synonyms': ['Lung']},\n",
       " {'name': 'Spleen', 'synonyms': ['Spleen']},\n",
       " {'name': 'Skin', 'synonyms': ['Skin']},\n",
       " {'name': 'Embryo', 'synonyms': ['Embryo']},\n",
       " {'name': 'Bone', 'synonyms': ['Bone']},\n",
       " {'name': 'Spinal Cord', 'synonyms': ['Spinal Cord']},\n",
       " {'name': 'Organoid', 'synonyms': ['Organoid']},\n",
       " {'name': 'Bladder', 'synonyms': ['Bladder']},\n",
       " {'name': 'Intestine', 'synonyms': ['Intestine']}]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview the generated vocabulary.\n",
    "Tissue.fetch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Specimen:Genes': \"['Wild Type']\"},\n",
       " {'Specimen:Genes': \"['Wnt1']\"},\n",
       " {'Specimen:Genes': \"['Wild Type']\"},\n",
       " {'Specimen:Genes': \"['Wild Type']\"},\n",
       " {'Specimen:Genes': \"['Aplnr']\"},\n",
       " {'Specimen:Genes': \"['Aplnr']\"},\n",
       " {'Specimen:Genes': \"['Wild Type']\"},\n",
       " {'Specimen:Genes': \"['Wild Type']\"},\n",
       " {'Specimen:Genes': \"['Gdf15']\"},\n",
       " {'Specimen:Genes': \"['Ct24']\"}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Next, we might want to take a look at the Specimen Genes\n",
    "gene_values = Scans.select(Specimens['Specimen:Genes']).fetch()\n",
    "gene_values[0:10] # print a sampling of the first 10 values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Step 7: Normalize the currently nested Specimen Genes\n",
    "import json\n",
    "\n",
    "def split_genes(s):\n",
    "    # first a custom function is needed to parse and split the values\n",
    "    if s:\n",
    "        # the encoding is not quite right so we fix the quotes, then parse\n",
    "        values = json.loads(s.replace(\"'\", '\"'))\n",
    "        for value in values:\n",
    "            # then yield each value\n",
    "            yield value\n",
    "\n",
    "# Unnest the Specimen Genes and create a vocabulary\n",
    "Specimen_Genes = Scans['Specimen:Genes'].to_atoms(unnest_fn=split_genes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Wild Type', 'synonyms': ['Wild Type']},\n",
       " {'name': 'Wnt1', 'synonyms': ['Wnt4', 'Wnt10B', 'Wnt1', 'Wnt11']},\n",
       " {'name': 'Aplnr', 'synonyms': ['Aplnr']},\n",
       " {'name': 'Gdf15', 'synonyms': ['Gdf15']},\n",
       " {'name': 'Ct24', 'synonyms': ['Ct24', 'Ct22']},\n",
       " {'name': 'Bleo', 'synonyms': ['Bleo']},\n",
       " {'name': 'Polycystin2', 'synonyms': ['Polycystin2']},\n",
       " {'name': 'Axin2', 'synonyms': ['Axin2']},\n",
       " {'name': 'Cited1', 'synonyms': ['Cited1']},\n",
       " {'name': 'Lgr5', 'synonyms': ['Lgr5']},\n",
       " {'name': 'Slc34a', 'synonyms': ['Slc34a']},\n",
       " {'name': 'Sox9', 'synonyms': ['Sox9']},\n",
       " {'name': 'Ndrg1', 'synonyms': ['Ndrg1']},\n",
       " {'name': 'C57Bl6', 'synonyms': ['C57Bl6', 'C57B6']},\n",
       " {'name': 'Blind', 'synonyms': ['Blind']},\n",
       " {'name': 'Myc', 'synonyms': ['Mycn', 'Myc']},\n",
       " {'name': 'Trpv4', 'synonyms': ['Trpv4']},\n",
       " {'name': 'Apela', 'synonyms': ['Apela']},\n",
       " {'name': 'Krt20', 'synonyms': ['Krt20']},\n",
       " {'name': 'Ihh', 'synonyms': ['Ihh', 'Shh']},\n",
       " {'name': 'Crlf1', 'synonyms': ['Crlf1']},\n",
       " {'name': 'Lfng', 'synonyms': ['Lfng']},\n",
       " {'name': 'Gfra3', 'synonyms': ['Gfra3']},\n",
       " {'name': 'Smek1', 'synonyms': ['Smek2', 'Smek1']},\n",
       " {'name': 'Ryk', 'synonyms': ['Ryk']},\n",
       " {'name': 'Six1', 'synonyms': ['Six1', 'Six2']},\n",
       " {'name': 'tOGL', 'synonyms': ['tOGL']},\n",
       " {'name': 'GM13398', 'synonyms': ['GM13398']},\n",
       " {'name': 'Admts18', 'synonyms': ['Adamts18', 'Admts18']},\n",
       " {'name': 'Foxd1', 'synonyms': ['Foxd1', 'Hoxd11']},\n",
       " {'name': 'Hs3st3a1', 'synonyms': ['Hs3st3a1']},\n",
       " {'name': 'Kdm2b', 'synonyms': ['Kdm2b']},\n",
       " {'name': 'Swiss Webster', 'synonyms': ['Swiss (Webster)', 'Swiss Webster']},\n",
       " {'name': 'rtTA', 'synonyms': ['rtTA']},\n",
       " {'name': 'tdt;Wnt11RFP', 'synonyms': ['tdt;Wnt11RFP']},\n",
       " {'name': 'Umod', 'synonyms': ['Umod']},\n",
       " {'name': 'Slc12a3', 'synonyms': ['Slc12a3']},\n",
       " {'name': 'Six2-eGFP Cited1-mCherry',\n",
       "  'synonyms': ['Six2-eGFP Cited1-mCherry']},\n",
       " {'name': 'Lrp4', 'synonyms': ['Lrp4']},\n",
       " {'name': 'R26', 'synonyms': ['R26']},\n",
       " {'name': 'Netrin1', 'synonyms': ['Netrin1']},\n",
       " {'name': 'AER;pTRE;Wnt11', 'synonyms': ['AER;pTRE;Wnt11']},\n",
       " {'name': 'AER;tetOMyc', 'synonyms': ['AER;tetOMyc']},\n",
       " {'name': 'Lkb1', 'synonyms': ['Lkb1']},\n",
       " {'name': 'AER;DTA', 'synonyms': ['AER;DTA']},\n",
       " {'name': 'Netrin1 conditional; Foxd1GC',\n",
       "  'synonyms': ['Netrin1 conditional; Foxd1GC']},\n",
       " {'name': 'MAFB-eGFP', 'synonyms': ['MAFB-eGFP']},\n",
       " {'name': 'Osx', 'synonyms': ['Osx']},\n",
       " {'name': 'Ctnnb1', 'synonyms': ['Ctnnb1']},\n",
       " {'name': 'Phf19', 'synonyms': ['Phf19']},\n",
       " {'name': 'Sunfish KO', 'synonyms': ['Sunfish KO']},\n",
       " {'name': 'Lyz2L10a', 'synonyms': ['Lyz2L10a']},\n",
       " {'name': 'JCK1423', 'synonyms': ['JCK1423']},\n",
       " {'name': 'Dlx5', 'synonyms': ['Dlx5']},\n",
       " {'name': 'Spp2', 'synonyms': ['Spp2']},\n",
       " {'name': 'Entpd5', 'synonyms': ['Entpd5']},\n",
       " {'name': 'Srd5a2', 'synonyms': ['Srd5a2']},\n",
       " {'name': 'Lyz2', 'synonyms': ['Lyz2']},\n",
       " {'name': 'Col2', 'synonyms': ['Col2']},\n",
       " {'name': 'Ntrk2', 'synonyms': ['Ntrk2']},\n",
       " {'name': 'Tinag1', 'synonyms': ['Tinag1']},\n",
       " {'name': 'Sp7', 'synonyms': ['Sp7']},\n",
       " {'name': 'Fzd2', 'synonyms': ['Fzd7', 'Fzd2']},\n",
       " {'name': 'Ror2c', 'synonyms': ['Ror2c']},\n",
       " {'name': 'Pax2', 'synonyms': ['Pax2']},\n",
       " {'name': 'Cdh5', 'synonyms': ['Cdh5']}]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 8: Create a vocabulary from the gene names\n",
    "Genes = Specimen_Genes['Specimen:Genes'].to_vocabulary()\n",
    "Genes.fetch() # preview the vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Step 9_:\n",
    "Clearly the vocabulary requires some tuning and human review. The default edit distance algorithm can be replaced with alternative matching algorithms. The algorithm did correctly find synonyms such as:\n",
    "`{'name': 'Swiss Webster', 'synonyms': ['Swiss Webster', 'Swiss (Webster)']},`\n",
    "But also incorrectly identified synonyms:\n",
    "`{'name': 'Six1', 'synonyms': ['Six1', 'Six2']},`\n",
    "For this reason, chisel allows alternative custom functions to be plugged in and used by the physical-level operators. Also, at this stage, the vocabulary can be reviewed by the domain user, who can correct the vocabulary and then load it into the database using chisel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Step 10 and beyond_: Several more transformations similar to the above would be performed before the schema evolution would be complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally, for demonstration purposes the relations are materialzed to a revised schema in a local catalog.\n",
    "local = chisel.connect('./catalog')\n",
    "with local.evolve():\n",
    "    local['.']['Scan.csv'] = RevisedScans\n",
    "    local['.']['Slide.csv'] = Slides\n",
    "    local['.']['Specimen.csv'] = Specimens\n",
    "    local['.']['Experiment.csv'] = Experiments\n",
    "    local['.']['Experiment_Type.csv'] = Experiment_Type\n",
    "    local['.']['Tissue.csv'] = Tissue\n",
    "    local['.']['Specimen_Gene.csv'] = Specimen_Genes\n",
    "    local['.']['Gene.csv'] = Genes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}