tutorials/00-FirstSteps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "939d9be3",
   "metadata": {},
   "source": [
    "# iBridges First Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c955adf",
   "metadata": {},
   "source": [
    "This note book is for the impatient user. We will take you through the main functionality of the iBridges API."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffb6d7d",
   "metadata": {},
   "source": [
    "To follow the notebook you need a valid iRODS configuration file installed in your `.irods` directory. If you do not have that please follow our guide [here](01-Setup-and-connect.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e02a7976",
   "metadata": {},
   "source": [
    "## Create an iRODS session (connection to iRODS server)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6fd51d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.interactive import interactive_auth\n",
    "session = interactive_auth()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26b76d89",
   "metadata": {},
   "source": [
    "The `interactive_auth` takes the default `.irods/irods_environment.json` to connect to the iRODS server. You can direct the authentication to a different file like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6d1ccf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "session = interactive_auth(irods_env_path=Path(\"~\").expanduser().joinpath(\".irods\", \"another_env_file.json\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb9577d",
   "metadata": {},
   "source": [
    "### Checking some session parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe178d2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(session.username)\n",
    "print(session.default_resc) # the resource to which data will be uploaded\n",
    "print(session.zone) # default home for iRODS /zone/home/username\n",
    "print(session.server_version)\n",
    "print(session.home)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e484823",
   "metadata": {},
   "source": [
    "## A word on iRODS paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a2f1607",
   "metadata": {},
   "source": [
    "On a local file name we have folders/directories and files. In iRODS we call them collections and data objects. In contrast to files, data objects carry system metadata and user defined metadata. Likewise for collections."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d5d770",
   "metadata": {},
   "source": [
    "`iBridges` supports iRODS paths with an own module `IrodsPath`. In this module all paths will be relative to the `session.home` unless they start with `/`.\n",
    "\n",
    "There are three ways to create an `IrodsPath` from your home."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15201901",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges import IrodsPath\n",
    "home = IrodsPath(session, session.home)\n",
    "home1 = IrodsPath(session, \"~\")\n",
    "home2 = IrodsPath(session)\n",
    "\n",
    "print(home)\n",
    "print(home1)\n",
    "print(home2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6db47a6",
   "metadata": {},
   "source": [
    "`IrodsPaths` implement some functionalities comparable to their counterparts in `Pathlib`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae669333",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some stats\n",
    "print(home)\n",
    "print(IrodsPath(session, home).exists())\n",
    "print(home.dataobject_exists())\n",
    "print(home.collection_exists())\n",
    "print(home.parent)\n",
    "\n",
    "# path manipulation\n",
    "print(home.joinpath(\"sub_collection\"))\n",
    "\n",
    "# iRODS system metadata\n",
    "print(f\"Size of {home}: {home.size}\") # size in bytes\n",
    "\n",
    "# iRODS operations\n",
    "new_coll = IrodsPath.create_collection(session, home.joinpath(\"new_coll_name\"))\n",
    "new_coll_path = IrodsPath(session, new_coll.path)\n",
    "print(f\"Created: {new_coll_path}\")\n",
    "new_coll_path = new_coll_path.rename(\"new_coll_name1\")\n",
    "print(f\"Moved to: {new_coll_path}\")\n",
    "new_coll_path.remove()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cddc08e",
   "metadata": {},
   "source": [
    "For a full Tutorial please have a look [here](02-iRODS-paths.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6716876",
   "metadata": {},
   "source": [
    "## Working with data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "236b368e",
   "metadata": {},
   "source": [
    "[Full Tutorial](03-Working-with-data.ipynb) and a tutorial for [Data Synchronisation](06-Data-sync.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f56d4204",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from ibridges.util import obj_replicas\n",
    "from ibridges import upload, download"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd1bd28",
   "metadata": {},
   "source": [
    "### Upload a file or folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec66ec0c",
   "metadata": {},
   "source": [
    "Create a local file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08d96d6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a local file\n",
    "local_path = Path.home().joinpath(\"demofile.txt\")\n",
    "f = open(local_path, \"a\")\n",
    "f.write(\"My content! Super important\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d9dee41",
   "metadata": {},
   "source": [
    "Determine the iRODS path the file should be uploaded to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "043536c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "irods_path = IrodsPath(session, session.home, \"new_coll\")\n",
    "if not irods_path.collection_exists():\n",
    "    coll = IrodsPath.create_collection(session, irods_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0de939a",
   "metadata": {},
   "source": [
    "We can first check that an upload would change with the `dry-run`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45991a0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload the file to our collection\n",
    "from pprint import pprint\n",
    "ops = upload(session, local_path, irods_path, dry_run=True, overwrite=True)\n",
    "ops.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8f827d3",
   "metadata": {},
   "source": [
    "The function will only show what really needs to be updated on iRODS based on checksums. Now we can really upload the file. We will use the `overwrite` flag to update all existing data in iRODS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb9768fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "changes = upload(session, local_path, irods_path, overwrite=True)\n",
    "changes.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "975d3b36",
   "metadata": {},
   "source": [
    "The `ops` dictionary can also be used to log what has been uploaded."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0893c4b",
   "metadata": {},
   "source": [
    "### Retrieve a  data object from iRODS and checking their status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6673452",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path = irods_path.joinpath(\"demofile.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f0d6e4",
   "metadata": {},
   "source": [
    "We can retrieve the system metadata of the object through its path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8da0670",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('data object name\\t', obj_path.name)\n",
    "print('data object path\\t', str(obj_path))\n",
    "print('data object size\\t', obj_path.size)\n",
    "print('data object checksum\\t', obj_path.checksum)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ba8f2b2",
   "metadata": {},
   "source": [
    "To retrieve the content of a data object or see on which storage resources the data is stored, we need to access the dataobject:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554065f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj = obj_path.dataobject\n",
    "print('data replicas\\t', obj_replicas(obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed47426",
   "metadata": {},
   "source": [
    "### Open a data object in read or write mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93288ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "stream = obj.open('r')\n",
    "text = stream.read().decode()\n",
    "stream.close()\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26bb46ff",
   "metadata": {},
   "source": [
    "### Download a data object or collection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5205a2",
   "metadata": {},
   "source": [
    "Downloads work similar as the uploads: use the `dry_run` to see what will change and use the `overwrite` flag to update all already existing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e734843b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ipath = IrodsPath(session, irods_path)\n",
    "print(ipath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54a1dd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "local_path = Path.home().joinpath(\"Downloads\")\n",
    "print(\"Download folder exists\", local_path.is_dir())\n",
    "local_changes = download(session, ipath, local_path, overwrite=True)\n",
    "local_changes.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1637df",
   "metadata": {},
   "source": [
    "## User defined metadata of data objects and collections"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d3ad69d",
   "metadata": {},
   "source": [
    "[Full Tutorial](04-Metadata.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "928bf82c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload a file\n",
    "local_path = Path.home().joinpath(\"demofile.txt\")\n",
    "irods_path = IrodsPath(session, session.home, \"new_coll\")\n",
    "if not irods_path.collection_exists():\n",
    "    coll = IrodsPath.create_collection(session, irods_path)\n",
    "changes = upload(session, local_path, irods_path, overwrite = True)\n",
    "changes.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "510e390a",
   "metadata": {},
   "source": [
    "### Retrieve an iRODS object or collection and list its metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ae7d602",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path = irods_path.joinpath(\"demofile.txt\")\n",
    "print(obj_path.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42007e1e",
   "metadata": {},
   "source": [
    "### View, add, set and delete metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfa7f559",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path.meta.add('NewKey', 'NewValue')\n",
    "obj_path.meta.add('NewKey', 'AnotherValue')\n",
    "print(obj_path.meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8aa144a",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path.meta.delete('NewKey', 'NewValue')\n",
    "print(obj_path.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b5cafa",
   "metadata": {},
   "source": [
    "We can also set the meta data to a single key, value, units pair. This will remove any other entries with the same key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "770cbec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path.meta.set(\"NewKey\", \"YetAnotherValue\")\n",
    "print(obj_path.meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2198d111-ee11-42e3-b24b-d774d95e7156",
   "metadata": {},
   "source": [
    "### Accessing metadata \n",
    "With the print function you can quickly inspect the metadata of an iRODS collection or object. If you want to extract and do something with the metadata, you can iterate over all metadata entries. We give a small example below where we assume that the metadata contains a key/name *AUTHOR*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8debeb07",
   "metadata": {},
   "outputs": [],
   "source": [
    "for md in obj_path.meta:\n",
    "    print(md.name, \"has value\", md.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215251fc",
   "metadata": {},
   "source": [
    "## Search for data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89009c88",
   "metadata": {},
   "source": [
    "How can you retrieve the location of an iRODS collection or data object by its metadata or partial part? To this end we provide the function `search`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52cb2500",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.search import search_data, MetaSearch\n",
    "\n",
    "#all collections and objects with \n",
    "result = search_data(session, metadata=MetaSearch(key=\"NewKey\", value=\"YetAnotherValue\"))\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aca8830",
   "metadata": {},
   "source": [
    "In the result we find the `IrodsPath` of the found collections and data objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861e4b95",
   "metadata": {},
   "source": [
    "If we are unsure about the exact metadata or collection or object name, we can use the wildcard `%`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e750edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search_data(session, session.home, path_pattern=\"new_coll/%\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a2d4a6",
   "metadata": {},
   "source": [
    "## Permissions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2dc3a91",
   "metadata": {},
   "source": [
    "**Note: In Yoda 1.9 users cannot set permissions any longer.**\n",
    "\n",
    "[Full Tutorial](05-Data-Sharing.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca96b65",
   "metadata": {},
   "source": [
    "### Accessing the permissions of a data object or collection in iRODS\n",
    "\n",
    "Objects and collections have permissions attached to them. Permissions, which work like access levels, must be specified per user or group. The basic permissions are `own` (implies reading and writing), `modify object` (editing and reading), and `read object`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eb3a627",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.permissions import Permissions\n",
    "\n",
    "# select a file to inspect and set permissions on\n",
    "item_path = IrodsPath(session, session.home) # Path to collection or data object\n",
    "collection = item_path.collection  # Get the collection, use item_path.dataobject for data objects\n",
    "\n",
    "# instantiate permissions with that object\n",
    "perm = Permissions(session, collection)\n",
    "print(f'Permissions for {item_path}:\\n')\n",
    "print(perm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2d1c093",
   "metadata": {},
   "source": [
    "### Available permissions on your iRODS server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "691dbaa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "perm.available_permissions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b84f75ba",
   "metadata": {},
   "source": [
    "### Adding permissions to a collection or data object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e2e1737",
   "metadata": {},
   "outputs": [],
   "source": [
    "perm.set('modify object', '<username or group name>')\n",
    "print(perm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5779c58",
   "metadata": {},
   "source": [
    "Note that some permission-types have synonyms:\n",
    "\n",
    "+ read object: 'read', 'read object', 'read_object'\n",
    "+ modify object: 'write', 'modify object', 'modify_object'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2ba10d",
   "metadata": {},
   "source": [
    "### Removing permissions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21bda746",
   "metadata": {},
   "outputs": [],
   "source": [
    "perm.set('null', '<username or group name>')\n",
    "print(perm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d52f16b",
   "metadata": {},
   "source": [
    "### Inheritance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76942f18",
   "metadata": {},
   "source": [
    "Collections have two special permissions level `inherit` and `noinherit`. From the point in time where inheritance in switched on, all newly added subcollections and data objects will inherit their initial permissions from the collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60251c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve a collection from iRODS\n",
    "coll_path = session.home\n",
    "coll = session.irods_session.collections.get(item_path)\n",
    "coll_perm = Permissions(session, coll)\n",
    "\n",
    "#Switch inheritance on\n",
    "coll_perm.set('inherit')\n",
    "print(coll_perm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec9bd025",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Switch inheritance off\n",
    "coll_perm.set('noinherit')\n",
    "print(coll_perm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79a02f14",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}