tutorials/02-iRODS-paths.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d6dbf987",
   "metadata": {},
   "source": [
    "# Small tutorial on iRODS paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfb329bc",
   "metadata": {},
   "source": [
    "* iRODS deals with POSIX-like paths, i.e. paths use the `/` as delimiter and all absolute paths start with `/`.\n",
    "* In default iRODS instances users have a personal home collection `/<zonename>/home/<username>`\n",
    "* In some iRODS instances like in Yoda instances, users are part of a group and only have access to group collections `/<yodazone>/home/<groupname>`\n",
    "\n",
    "In *iBridges* we allow all paths by default to be strings, which have the to be formatted as described above, i.e. for up and downloads all source and destination paths can be offered as `str`.\n",
    "However, we also offer a small class `IrodsPath` with which some formatting is done automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d4a113",
   "metadata": {},
   "source": [
    "## The iRODS home"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "139a2c8b",
   "metadata": {},
   "source": [
    "In your configuration file `irods_environment.json` you can set a default working directory on the iRODS instance with:\n",
    "```\n",
    "\"irods_home\": \"/<zone>/home/<user or groupname>\"\n",
    "```\n",
    "This information is stored in the iBridges session object and employed by the iBridges IrodsPath object."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd887f2",
   "metadata": {},
   "source": [
    "### Create a session and set your iRODS default collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de207d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.interactive import interactive_auth\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "648f8c49",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = interactive_auth()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7bd4841",
   "metadata": {},
   "outputs": [],
   "source": [
    "session.home"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c6f8a9",
   "metadata": {},
   "source": [
    "There are three ways to set the `irods_home`:\n",
    "\n",
    "1. You can set the \"irods_home\" in the configuration file `irods_environment.json`\n",
    "2. You can pass it as a parameter when creating the session\n",
    "3. You can set it later by `session.home = <YOUR_IRODS_PATH>`\n",
    "\n",
    "If none of the options are used, the `session` will set it automatically to `/<zonename>/home/<username>`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e27d67b7",
   "metadata": {},
   "source": [
    "**Note**, that it is not verified that this path really exists on the iRODS server. It is merely a configuration item."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d20dd782",
   "metadata": {},
   "source": [
    "### Verify that your current working directory exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edc53ead",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges import IrodsPath\n",
    "home_path = IrodsPath(session, session.home)\n",
    "print(home_path)\n",
    "print(f\"Path exists: {home_path.exists()}\")\n",
    "print(f\"Path is a collection: {home_path.collection_exists()}\")\n",
    "print(f\"Path is a data object: {home_path.dataobject_exists()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb70b77",
   "metadata": {},
   "source": [
    "You can address your iRODS home in the following ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5aba574",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(IrodsPath(session))\n",
    "print(IrodsPath(session, session.home))\n",
    "print(IrodsPath(session, \"~\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8fa13a",
   "metadata": {},
   "source": [
    "## iRODS path manipulations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f925268e",
   "metadata": {},
   "source": [
    "###  Create an IrodsPath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33ea8da2",
   "metadata": {},
   "outputs": [],
   "source": [
    "irods_path = IrodsPath(session, 'my_study', 'my_experiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b99d65b",
   "metadata": {},
   "source": [
    "Let's inspect the properties and functions of the empty iRODS path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f770f351",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(irods_path)\n",
    "print(irods_path.absolute())\n",
    "print(irods_path.parts)\n",
    "print(irods_path.parent)\n",
    "print(irods_path.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d97efb0f",
   "metadata": {},
   "source": [
    "When we create a new iRODS path which does not start with `/` the `IrodsPath` will automatically expand the path with the current working directory, i.e the iRODS home.\n",
    "**Let's see how to omit the automatic expansion of the path with your `session.home`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1da4f2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ipath = IrodsPath(session, \"/\", \"some\", \"other\", \"irods_coll\")\n",
    "print(ipath)\n",
    "print(ipath.absolute())\n",
    "print(ipath.parts)\n",
    "print(ipath.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f3a284",
   "metadata": {},
   "source": [
    "**Note,** that creating an `IrodsPath` instance does not automatically create the collection or data object on the iRODS server! To create the collection do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9591b28a",
   "metadata": {},
   "outputs": [],
   "source": [
    "IrodsPath.create_collection(session, irods_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6dccc6",
   "metadata": {},
   "source": [
    "### Extending an iRODS path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c374a6ae",
   "metadata": {},
   "source": [
    "Assume we would like to create a new path in iRODS e.g. to upload data to or to create a new collection. We want to extend our `home_path` with `testdata/experiment1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04c0e7d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_loc = home_path.joinpath('testdata', 'experiment1')\n",
    "print(new_loc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "453129ab",
   "metadata": {},
   "source": [
    "### Joining iRODS and Windows paths\n",
    "Assume you are working on windows and would like to create a copy of a local working folder on iRODS. Windows uses a different parts separator `\\`. In such a case we advise you to work with the `pathlib.Path` or the `pathlib.WindowsPath`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d0dab96",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import PureWindowsPath\n",
    "win_path = PureWindowsPath('windows','path','to', 'study', 'experiment2')\n",
    "print(win_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7671dd16",
   "metadata": {},
   "source": [
    "We can assemble the new iRODS path `study/experiment2` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3a4e653",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(win_path.parts[-2:])\n",
    "new_loc = home_path.joinpath(win_path.parts[-2], win_path.parts[-1])\n",
    "print(new_loc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4c6f2f",
   "metadata": {},
   "source": [
    "Similarly we can extend a Windows path with a relative iRODS path: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c547ef14",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(irods_path)\n",
    "new_loc = win_path.joinpath(irods_path.parts[0], irods_path.parts[1])\n",
    "print(new_loc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cb2a6e0",
   "metadata": {},
   "source": [
    "Other useful functions for path manipulations are:\n",
    "- `IrodsPath.parent`, which will give you the direct parent of a path\n",
    "- `IrodsPath.relative_to`, which computes a new path relative to another path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c29be5fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "irods_path.relative_to(IrodsPath(session, \"~\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70437f5b",
   "metadata": {},
   "source": [
    "## Functions to change collections and data object on the iRODS server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dfe6245",
   "metadata": {},
   "source": [
    "In the previous section we saw how to create iRODS paths and how to verify them. However, we have not yet changed something on the iRODS server. In this section we will go through some functionality to:\n",
    "\n",
    "- Create a new collection from a path\n",
    "- Retrieve properties of collections and data objects\n",
    "- Get the data object or collection from iRODS by its path\n",
    "- Rename/move data objects and collections\n",
    "- Delete a data object or collection from its path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec09acb",
   "metadata": {},
   "source": [
    "### Create a new collection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c526a32f",
   "metadata": {},
   "source": [
    "Let's have a look how to create a new subcollection in our iRODS home. We will make sure that `irods_path` does not already exist on the iRODS server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c10f35a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(irods_path)\n",
    "print(f\"Already exists: {irods_path.exists()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f107d2c",
   "metadata": {},
   "source": [
    "So let us create this location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d45e354",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = IrodsPath.create_collection(session, irods_path)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae806474",
   "metadata": {},
   "source": [
    "The command above will create the whole subcollection tree if it does not exist yet and it will return the python object `out` which is of type `iRODSCollection`. In [02-Working-with-data.ipynb](02-Working-with-data.ipynb) we will show you what you can do with such a collection object."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0783525",
   "metadata": {},
   "source": [
    "### Rename or move a collection or data object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e131784d",
   "metadata": {},
   "source": [
    "If we are not happy with the name of our new collection, we can change that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76cebbb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(irods_path.exists())\n",
    "new_path = irods_path.rename(\"my_cool_experiment\")\n",
    "print(irods_path.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af5283a",
   "metadata": {},
   "source": [
    "You see, that `irods_path` will stay the same, but will no longer point to an existing collection. The new location is returned as an `IrodsPath`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f1de951",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d275ecf",
   "metadata": {},
   "source": [
    "We can use the same function to move the collection to a different part of the iRODS tree, even if it does not exist yet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48601173",
   "metadata": {},
   "outputs": [],
   "source": [
    "loc = new_path.rename(IrodsPath(session, \"some\", \"coll\", \"my_cool_experiment\"))\n",
    "loc.exists()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e92f9d05",
   "metadata": {},
   "source": [
    "### Delete a collection or data object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52519932",
   "metadata": {},
   "source": [
    "If you did not loose overview over where our new collection is, you can delete it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403cd9f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(loc)\n",
    "loc.exists()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2356f493",
   "metadata": {},
   "outputs": [],
   "source": [
    "loc.remove()\n",
    "print(loc.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7d7049",
   "metadata": {},
   "source": [
    "**Note**, again the instance of `IrodsPath` will continue to exist, while the location behind the path has been deleted from the iRODS instance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74250a68",
   "metadata": {},
   "source": [
    "### Size and checksum of the data object or collection behind a path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5412ec6b",
   "metadata": {},
   "source": [
    "With `IrodsPath` we do not necessarily have to retrieve explicitly the data object or collection; `IrodsPath` contains some functions to get more information on the item behind the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cf0850f",
   "metadata": {},
   "outputs": [],
   "source": [
    "irods_path = IrodsPath(session)\n",
    "print(irods_path.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ae2a245",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}