tutorials/04-Metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f9e6811a",
   "metadata": {},
   "source": [
    "# User-defined metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8564525b",
   "metadata": {},
   "source": [
    "Next to system metadata, iRODS allows you to create own metadata with data objects and collections.\n",
    "\n",
    "You can use that metadata to describe your data and later search for this data; and it can help you keeping the overview of what was the input for an analysis and what is the outcome.\n",
    "\n",
    "<img src=\"img/DataObject5.png\" width=\"400\">\n",
    "\n",
    "Technically, iRODS offers metadata as key-value-units triple. Let's investigate this:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde731e4",
   "metadata": {},
   "source": [
    "## Add metadata to data objects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "784fdac1",
   "metadata": {},
   "source": [
    "As always: first we have to create an iRODS session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277459df",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.interactive import interactive_auth\n",
    "session = interactive_auth()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb0524a2",
   "metadata": {},
   "source": [
    "Make sure we have our *demo* collection and file available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19ad51aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.path import IrodsPath\n",
    "\n",
    "irods_path = IrodsPath(session, '~')\n",
    "print(\"Current working location:\", irods_path)\n",
    "irods_coll_path = irods_path.joinpath('demo')\n",
    "print(\"New collection name:\", irods_coll_path)\n",
    "coll = IrodsPath.create_collection(session, irods_coll_path)\n",
    "print(\"New collection is created:\", irods_coll_path.collection_exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81094d42",
   "metadata": {},
   "source": [
    "Now we can retrieve a data object and insect its metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69066865",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.path import IrodsPath\n",
    "\n",
    "irods_coll_path = IrodsPath(session, '~').joinpath('demo')\n",
    "obj = irods_coll_path.joinpath('demofile.txt')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d9a578b",
   "metadata": {},
   "source": [
    "Most probably you will see no metadata in the above cell. **Note, that system metadata and user-defined metadata are two different entities in a data object!**\n",
    "With the command `MetaData(obj)` we only retrieve the user-defined metadata.\n",
    "\n",
    "<img src=\"img/DataObject4.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fb67574",
   "metadata": {},
   "source": [
    "Now we can add some own metadata. The metadata comes as key-value-units triple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79a1bb89",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.add('Key', 'Value', 'Units')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafbeaff",
   "metadata": {},
   "source": [
    "Sometimes we do not really have `units`, so we can leave this part empty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb439f38",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.add('Author', 'Christine')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bfbfa71",
   "metadata": {},
   "source": [
    "We can also add a second author:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb35166a",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.add('Author', 'Raoul')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "116c0f10",
   "metadata": {},
   "source": [
    "You see, that keys in **iRODS metadata keys can have different values**. That is different from python dictionaries where one key can only have one value. **How then to overwrite a value?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf6d7f93",
   "metadata": {},
   "source": [
    "## Overwrite metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5b7fef",
   "metadata": {},
   "source": [
    "If you wish to *overwrite* a value, you will first have to remove the old metadata and subsequently add a new metadata entry. **NOTE, that all entries with the key will be deleted.** If you want to be more specific you will need to give the value and the units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0cbd9bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.delete('Author')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3348ce2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.add('Author', 'Raoul')\n",
    "obj.meta.add('Author', 'Christine')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c0eaf42",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.delete('Author', 'Christine')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f47a59",
   "metadata": {},
   "source": [
    "You can also set all existing values to **one** new value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f0a95b",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.set('Author', 'Maarten')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af4d99f",
   "metadata": {},
   "source": [
    "iRODS metadata also has a an entry called `units`. The same principles that we showed above, i.e. having the same key-value pair with several units, deleting and setting values, apply to units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c8ee90c",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.add('key', 'value', 'units1')\n",
    "obj.meta.add('key', 'value', 'units2')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d39bf4eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.set('key', 'value', 'units3')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97b93549",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj.meta.delete('key', 'value')\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3f54857",
   "metadata": {},
   "source": [
    "## Add metadata to collections"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51438d57",
   "metadata": {},
   "source": [
    "The same functionality we saw above, we can use for collections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65fb35a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "coll = irods_coll_path\n",
    "print(coll.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2af31459",
   "metadata": {},
   "outputs": [],
   "source": [
    "coll.meta.add('TypeOfCollection', 'Results')\n",
    "print(coll.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7eacaba",
   "metadata": {},
   "source": [
    "## Which metadata can help you keeping an overview?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb48bb74",
   "metadata": {},
   "source": [
    "iRODS metadata can help you keeping an overview while you are working with data and maybe many files which have relations to each other. There are ontologies which define keywords and links between keywords like the **[prov-o Ontology](https://www.w3.org/TR/prov-o/#prov-o-at-a-glance)**.\n",
    "\n",
    "Let's see how we can annotate our test data, so that we know that it is test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbadefc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "coll.meta.add('prov:wasGeneratedBy', 'Christine')\n",
    "coll.meta.add('CollectionType', 'testcollection')\n",
    "obj.meta.add('prov:SoftwareAgent', 'iRODS jupyter Tutorial')\n",
    "obj.meta.add('prov:wasGeneratedBy', 'Maarten')\n",
    "obj.meta.add('DataType', 'testdata')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be8180b",
   "metadata": {},
   "source": [
    "Now we have some more descriptive metadata that gives us hints, in which context the data was created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0b8e2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(coll.meta)\n",
    "print()\n",
    "print(obj.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f721760",
   "metadata": {},
   "source": [
    "## Finding data by their metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59518813",
   "metadata": {},
   "source": [
    "Metadata does not only help you to keep an overview over your data, but can also be used to select and retrieve data. In iBridges you can use the user-defined metadata and some system metadata fields to search for data.\n",
    "\n",
    "In our first example, we are looking for objects and collections called *demo* in our `home`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6918fe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.search import search_data, MetaSearch\n",
    "result = search_data(session, path=session.home, path_pattern=\"demo\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea7129f8",
   "metadata": {},
   "source": [
    "The output is a list of IrodsPath's indicating the locations of the data objects and collections.\n",
    "If no `path` is provided, *ibridges* will automatically fall back on your `home`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b054794",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, metadata=MetaSearch(key='prov:wasGeneratedBy', value='Christine'))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81d73352",
   "metadata": {},
   "source": [
    "If we do not want to specify the particular value for this metadata entry, we can leave it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49a9f17e",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, metadata=MetaSearch(key='prov:wasGeneratedBy'))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69a2e046",
   "metadata": {},
   "source": [
    "Now we also receive the data object that was generated by *Maarten*.\n",
    "\n",
    "And of course we can combine information about the path and the metadata. they will be connected with `and`. The following search will retrieve all data objects and collections wich are labeled with a metadata key *'prov:wasGeneratedBy'* and whose path has the prefix */nluu12p/home/research-test-christine/demo/*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0d2fad9",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=IrodsPath(session, session.home, 'demo'),\n",
    "                     metadata=MetaSearch(key='prov:wasGeneratedBy'))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8750ba85",
   "metadata": {},
   "source": [
    "## Searches using wildcards"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ec06eaa",
   "metadata": {},
   "source": [
    "Sometimes we are not sure about the exact pattern that we search for, be it metadata keys, values and units or path patterns. iRODS knows the `%` sign as a wild card."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d26000f",
   "metadata": {},
   "source": [
    "### Wildcards in metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a7b1fe9",
   "metadata": {},
   "source": [
    "Assume we know that some data was annotated according to the Prov-O Ontology and its abbrviation is `prov`, but we do not know which terms of that Ontology was used. In such a case we can find all metadata annotated with a key with the prefix `prov:` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e19e205c",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=IrodsPath(session, session.home),\n",
    "                     metadata=MetaSearch(key='prov:%'))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea29f04f",
   "metadata": {},
   "source": [
    "### Wildcards in path patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efc56727",
   "metadata": {},
   "source": [
    "Let us go back to the very first example of this section, we are loking for collections and data objects called `demo` and they need to lie directly in our `home`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c5636d",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=session.home, path_pattern=\"demo\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edef8def",
   "metadata": {},
   "source": [
    "How can we retrieve all `demo` collections and objects even if they are or lie in subcollections? Lets first create subcollections in `demo` called `demo` and `demo1`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1436239",
   "metadata": {},
   "outputs": [],
   "source": [
    "irods_path = IrodsPath(session, \"demo\", \"demo\")\n",
    "print(irods_path)\n",
    "IrodsPath.create_collection(session, irods_path)\n",
    "irods_path = IrodsPath(session, \"demo\", \"demo1\")\n",
    "print(irods_path)\n",
    "IrodsPath.create_collection(session, irods_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd5849c",
   "metadata": {},
   "source": [
    "Now let's see how to use the wildcard to find those two collections."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776e36f0",
   "metadata": {},
   "source": [
    "#### 1. Find all data and collections ending with `demo`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2354ae2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=session.home, path_pattern=\"%demo\")\n",
    "print('\\n'.join([str(p) for p in result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c710e35",
   "metadata": {},
   "source": [
    "#### 2. Find all data and collections starting with `demo`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e062cbf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=session.home, path_pattern=\"demo%\")\n",
    "print('\\n'.join([str(p) for p in result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05958fd7",
   "metadata": {},
   "source": [
    "#### 3. Find all collections and data called `demo` that on the 5th layer of the collection tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26cbab9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, path=session.home, path_pattern=\"%/%/%/%/%/demo\")\n",
    "print('\\n'.join([str(p) for p in result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4972990d",
   "metadata": {},
   "source": [
    "#### 4. Find all `txt` files that lie on a collection path that contains `demo`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d1d9ff",
   "metadata": {},
   "source": [
    "For this case we have to think of a pattern for the collection path and the object name and separate both with `/`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6781e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "coll_pattern = \"%demo%\"\n",
    "obj_pattern = \"%.txt\"\n",
    "print(f\"Search pattern: {coll_pattern+'/'+obj_pattern}\")\n",
    "result = search_data(session, path=session.home, path_pattern=coll_pattern+\"/\"+obj_pattern)\n",
    "print('\\n'.join([str(p) for p in result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14bc70b0",
   "metadata": {},
   "source": [
    "## Retrieving data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "946e355c",
   "metadata": {},
   "source": [
    "Now that we have the search results we can use the `IrodsPath` to download them or to fetch more information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff72b0ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(result[0].size)\n",
    "print(result[0].collection_exists())\n",
    "print(result[0].dataobject_exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4072812e",
   "metadata": {},
   "source": [
    "# Metadata archives\n",
    "\n",
    "In most cases the user is encouraged to access and manipulate metadata through the `MetaData` class. However, there are some cases where it can be useful to create an archive of all metadata in a collection and all subcollections and data objects. One example might be a backup of the data and metadata on a system that does not support metadata. Another might be to easily transfer metadata from one iRODS system to another. A final use case might be having access to the metadata during computation on a system that is not connected to the internet."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b343610",
   "metadata": {},
   "source": [
    "## Creating a metadata archive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a92e519",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.data_operations import create_meta_archive\n",
    "\n",
    "collection_path = IrodsPath(session, \"demo\")\n",
    "create_meta_archive(session, collection_path, \"meta_archive.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c20eaee",
   "metadata": {},
   "source": [
    "This creates a file \"meta_archive.json\" in your current local directory of this jupyter notebook which contains all metadata of all subcollections and data objects in this collection \"demo\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c378d37",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat meta_archive.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00d0e863",
   "metadata": {},
   "source": [
    "## Applying a metadata archive\n",
    "\n",
    "This restores/overwrites the metadata on the iRODS server with the metadata from the archive. Make sure that the paths of the subcollections and data objects have not changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c49162e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.data_operations import apply_meta_archive\n",
    "\n",
    "apply_meta_archive(session, \"meta_archive.json\", collection_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}