examples/2-condition-to-imaging-study.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Let us assume that we are given a CSV or Excel file that contains some information that we should\n",
    " query on our FHIR server. In the `example.csv` file we have a list of ICD 10 codes, but it may\n",
    " also be a list of patient IDs or anything else.\n",
    "\n",
    "The process:\n",
    " 1. Find all patients with these conditions.\n",
    " 2. Collect all of their CTs.\n",
    " 3. Download the CTs.\n",
    "\n",
    "We have created this example according to our FHIR server, where we also can download studies\n",
    "using the Dicom Web Adapter. For this purpose we use the [DicomWebClient](https://dicomweb-client.readthedocs.io/en/latest/usage.html).\n",
    "I am not sure on how this works at other university hospitals, so if\n",
    " you have any particular requests for this, please do not hesitate and ask.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [
    {
     "data": {
      "text/plain": "  icd_10\n0  C78.7\n1  C24.9",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>icd_10</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>C24.9</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "condition_df = pd.read_csv(\"example.csv\")\n",
    "condition_df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now do our call, and we get all the Conditions that have this ICD-10 code.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query & Build DF (Condition): 100%|██████████| 2/2 [00:00<00:00,  2.02it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": "        patient_id icd_10\n0    <redacted-id>  C78.7\n1    <redacted-id>  C78.7\n2    <redacted-id>  C78.7\n3    <redacted-id>  C78.7\n4    <redacted-id>  C78.7\n..             ...    ...\n195  <redacted-id>  C24.9\n196  <redacted-id>  C24.9\n197  <redacted-id>  C24.9\n198  <redacted-id>  C24.9\n199  <redacted-id>  C24.9\n\n[200 rows x 2 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>patient_id</th>\n      <th>icd_10</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C78.7</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>195</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C24.9</td>\n    </tr>\n    <tr>\n      <th>196</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C24.9</td>\n    </tr>\n    <tr>\n      <th>197</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C24.9</td>\n    </tr>\n    <tr>\n      <th>198</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C24.9</td>\n    </tr>\n    <tr>\n      <th>199</th>\n      <td>&lt;redacted-id&gt;</td>\n      <td>C24.9</td>\n    </tr>\n  </tbody>\n</table>\n<p>200 rows × 2 columns</p>\n</div>"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fhir_pyrate import Ahoy, Pirate, DicomDownloader\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "# I have stored these variables in an .env file\n",
    "BASIC_AUTH = os.environ[\"BASIC_AUTH\"]\n",
    "REFRESH_AUTH = os.environ[\"REFRESH_AUTH\"]\n",
    "SEARCH_URL = os.environ[\"SEARCH_URL\"]\n",
    "DICOM_WEB_URL = os.environ[\"DICOM_WEB_URL\"]\n",
    "\n",
    "# Init authentication\n",
    "auth = Ahoy(\n",
    "    auth_type=\"token\",\n",
    "    auth_method=\"env\",\n",
    "    auth_url=BASIC_AUTH,\n",
    "    refresh_url=REFRESH_AUTH,\n",
    ")\n",
    "search = Pirate(\n",
    "    auth=auth, base_url=SEARCH_URL, print_request_url=False, num_processes=1\n",
    ")\n",
    "condition_patient_df = search.trade_rows_for_dataframe(  # Runs the patient queries in parallel\n",
    "    df=condition_df,\n",
    "    resource_type=\"Condition\",\n",
    "    request_params={\n",
    "        \"_sort\": \"_id\",\n",
    "        \"_count\": 100,\n",
    "    },\n",
    "    with_ref=True,\n",
    "    df_constraints={\"code\": (\"http://fhir.de/CodeSystem/dimdi/icd-10-gm%7C\", \"icd_10\")},\n",
    "    fhir_paths=[(\"patient_id\", \"subject.reference\")],\n",
    "    num_pages=1,  # This option only collects the results from the first page,\n",
    "    # which depends on the value set for count\n",
    ")\n",
    "redacted_df = condition_patient_df.copy()\n",
    "redacted_df[\"patient_id\"] = \"<redacted-id>\"\n",
    "redacted_df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, we take the patients that we have found through Condition, and we make sure that there are no\n",
    "duplicates."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "157"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "patient_df = pd.DataFrame(\n",
    "    condition_patient_df[\"patient_id\"].drop_duplicates(keep=\"first\")\n",
    ")\n",
    "len(patient_df)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We then take the patients and look for their ImagingStudies, where we also decide what kind of\n",
    "arguments we should to store.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query & Build DF (ImagingStudy): 100%|██████████| 2/2 [00:00<00:00,  6.08it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": "                          started modality procedureCode  \\\n0   2018-01-31T11:13:26.000+01:00       CT           CTT   \n0   2018-04-13T10:25:37.000+02:00       CT           CTT   \n0   2019-07-26T10:42:38.000+02:00       CT           CTT   \n0   2018-09-19T10:35:20.000+02:00       CT           CTT   \n0   2016-02-24T10:08:50.000+01:00       CT           CTT   \n..                            ...      ...           ...   \n55  2018-04-13T10:25:37.000+02:00       PT         PCTGK   \n55  2013-12-12T09:23:36.000+01:00       PT         PCTGK   \n55  2012-08-01T12:19:14.000+02:00       PT         PCTGK   \n55  2020-01-20T10:27:57.000+01:00       PT         PCTGK   \n55  2017-02-02T11:40:25.000+01:00       PT         PCTGK   \n\n                                   study_instance_uid  \\\n0   1.2.826.0.1.3680043.8.498.24222694654806877939...   \n0   1.2.826.0.1.3680043.8.498.24222694654806877939...   \n0   1.2.826.0.1.3680043.8.498.24222694654806877939...   \n0   1.2.826.0.1.3680043.8.498.24222694654806877939...   \n0   1.2.826.0.1.3680043.8.498.24222694654806877939...   \n..                                                ...   \n55  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n55  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n55  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n55  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n55  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n\n                                  series_instance_uid series_code  \\\n0   1.2.826.0.1.3680043.8.498.33463995182843850024...          PR   \n0   1.2.826.0.1.3680043.8.498.33463995182843850024...          PR   \n0   1.2.826.0.1.3680043.8.498.33463995182843850024...          PR   \n0   1.2.826.0.1.3680043.8.498.33463995182843850024...          PR   \n0   1.2.826.0.1.3680043.8.498.33463995182843850024...          CT   \n..                                                ...         ...   \n55  1.2.826.0.1.3680043.8.498.33463995182843850024...          CT   \n55  1.2.826.0.1.3680043.8.498.33463995182843850024...          CT   \n55  1.2.826.0.1.3680043.8.498.33463995182843850024...          PT   \n55  1.2.826.0.1.3680043.8.498.33463995182843850024...          PT   \n55  1.2.826.0.1.3680043.8.498.33463995182843850024...          PT   \n\n   numberOfInstances     patient_id  \n0                  2  <redacted-id>  \n0                  2  <redacted-id>  \n0                  2  <redacted-id>  \n0                  2  <redacted-id>  \n0                  2  <redacted-id>  \n..               ...            ...  \n55               199  <redacted-id>  \n55                 1  <redacted-id>  \n55                 3  <redacted-id>  \n55               244  <redacted-id>  \n55               244  <redacted-id>  \n\n[331 rows x 8 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>started</th>\n      <th>modality</th>\n      <th>procedureCode</th>\n      <th>study_instance_uid</th>\n      <th>series_instance_uid</th>\n      <th>series_code</th>\n      <th>numberOfInstances</th>\n      <th>patient_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2018-01-31T11:13:26.000+01:00</td>\n      <td>CT</td>\n      <td>CTT</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PR</td>\n      <td>2</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>2018-04-13T10:25:37.000+02:00</td>\n      <td>CT</td>\n      <td>CTT</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PR</td>\n      <td>2</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>2019-07-26T10:42:38.000+02:00</td>\n      <td>CT</td>\n      <td>CTT</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PR</td>\n      <td>2</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>2018-09-19T10:35:20.000+02:00</td>\n      <td>CT</td>\n      <td>CTT</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PR</td>\n      <td>2</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>2016-02-24T10:08:50.000+01:00</td>\n      <td>CT</td>\n      <td>CTT</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>CT</td>\n      <td>2</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2018-04-13T10:25:37.000+02:00</td>\n      <td>PT</td>\n      <td>PCTGK</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>CT</td>\n      <td>199</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2013-12-12T09:23:36.000+01:00</td>\n      <td>PT</td>\n      <td>PCTGK</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>CT</td>\n      <td>1</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2012-08-01T12:19:14.000+02:00</td>\n      <td>PT</td>\n      <td>PCTGK</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PT</td>\n      <td>3</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2020-01-20T10:27:57.000+01:00</td>\n      <td>PT</td>\n      <td>PCTGK</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PT</td>\n      <td>244</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n    <tr>\n      <th>55</th>\n      <td>2017-02-02T11:40:25.000+01:00</td>\n      <td>PT</td>\n      <td>PCTGK</td>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>PT</td>\n      <td>244</td>\n      <td>&lt;redacted-id&gt;</td>\n    </tr>\n  </tbody>\n</table>\n<p>331 rows × 8 columns</p>\n</div>"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imaging_df = search.trade_rows_for_dataframe(  # Runs the patient queries in parallel with inputs\n",
    "    # from the given DataFrame\n",
    "    df=patient_df.head(\n",
    "        2\n",
    "    ),  # We only download for 2 patients to keep the DataFrame small for the visualization\n",
    "    resource_type=\"ImagingStudy\",\n",
    "    request_params={\n",
    "        \"_sort\": \"_id\",\n",
    "        \"modality\": \"CT\",\n",
    "        \"_count\": 100,\n",
    "    },\n",
    "    df_constraints={\n",
    "        \"subject\": \"patient_id\",\n",
    "    },\n",
    "    with_ref=True,\n",
    "    fhir_paths=[\n",
    "        \"started\",\n",
    "        (\"modality\", \"modality.code\"),\n",
    "        (\"procedureCode\", \"procedureCode.coding.code\"),\n",
    "        (\"study_instance_uid\", \"identifier.where(system = 'urn:dicom:uid').value.replace('urn:oid:', '')\"),\n",
    "        (\"series_instance_uid\", \"series.uid\"),\n",
    "        (\"series_code\", \"series.modality.code\"),\n",
    "        (\"numberOfInstances\", \"series.numberOfInstances\"),\n",
    "    ],\n",
    ")\n",
    "# The series are currently stored in a list\n",
    "imaging_df = imaging_df.explode(\n",
    "    [\n",
    "        \"series_instance_uid\",\n",
    "        \"series_code\",\n",
    "        \"numberOfInstances\",\n",
    "    ]\n",
    ")\n",
    "import numpy as np\n",
    "import pydicom\n",
    "\n",
    "redacted_df = imaging_df.copy()\n",
    "redacted_df[\"patient_id\"] = \"<redacted-id>\"\n",
    "redacted_df[\"started\"] = np.random.permutation(redacted_df[\"started\"].values)\n",
    "redacted_df[\"study_instance_uid\"] = pydicom.uid.generate_uid(entropy_srcs=[\"salt\"])\n",
    "redacted_df[\"series_instance_uid\"] = pydicom.uid.generate_uid(entropy_srcs=[\"pepper\"])\n",
    "redacted_df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Filter the series by CT."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "filtered = imaging_df.loc[imaging_df[\"series_code\"] == \"CT\"]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Download the studies using the DicomDownloader, which needs a specific URL pointing to the\n",
    "DicomWebAdapter and that specify the PACS that we want to use."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading Rows:   0%|          | 0/1 [00:00<?, ?it/s]\n",
      "Downloading Instance: 0it [00:00, ?it/s]\u001B[A\n",
      "Downloading Instance: 1it [00:00,  1.09it/s]\u001B[A\n",
      "Downloading Instance: 2it [00:01,  2.00it/s]\u001B[A\n",
      "Downloading Rows: 100%|██████████| 1/1 [00:01<00:00,  1.19s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": "Empty DataFrame\nColumns: []\nIndex: []",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "downloader = DicomDownloader(\n",
    "    auth=auth, dicom_web_url=DICOM_WEB_URL, output_format=\"nifti\"\n",
    ")\n",
    "\n",
    "successful_df, error_df = downloader.download_data_from_dataframe(\n",
    "    filtered.head(1),  # Download only the last element, just for testing\n",
    "    output_dir=\"out\",\n",
    "    study_uid_col=\"study_instance_uid\",\n",
    "    series_uid_col=\"series_instance_uid\",\n",
    "    download_full_study=False,\n",
    ")\n",
    "error_df"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The error DataFrame is empty, which means that no errors have occured. We can then check the\n",
    "successful downloads to find out in which folder the series was stored."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "                                  study_instance_uid  \\\n0  1.2.826.0.1.3680043.8.498.24222694654806877939...   \n\n                                 series_instance_uid  \\\n0  1.2.826.0.1.3680043.8.498.33463995182843850024...   \n\n                deidentified_study_instance_uid  \\\n0  2.25.236507833011739959424223550367290629720   \n\n               deidentified_series_instance_uid  \\\n0  2.25.201217751562255229154918195968192653161   \n\n                                         download_id  \\\n0  65425e1330b0895737d07d0fa29cd2614ff5dd9c7b566c...   \n\n                                       download_path  \n0  out/6c249114e1ebfc8c38b812efdf3ba859a1f57d56bb...  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>study_instance_uid</th>\n      <th>series_instance_uid</th>\n      <th>deidentified_study_instance_uid</th>\n      <th>deidentified_series_instance_uid</th>\n      <th>download_id</th>\n      <th>download_path</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1.2.826.0.1.3680043.8.498.24222694654806877939...</td>\n      <td>1.2.826.0.1.3680043.8.498.33463995182843850024...</td>\n      <td>2.25.236507833011739959424223550367290629720</td>\n      <td>2.25.201217751562255229154918195968192653161</td>\n      <td>65425e1330b0895737d07d0fa29cd2614ff5dd9c7b566c...</td>\n      <td>out/6c249114e1ebfc8c38b812efdf3ba859a1f57d56bb...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dummy values\n",
    "import hashlib\n",
    "\n",
    "redacted_df = successful_df.copy()\n",
    "redacted_df[\"study_instance_uid\"] = pydicom.uid.generate_uid(entropy_srcs=[\"salt\"])\n",
    "redacted_df[\"series_instance_uid\"] = pydicom.uid.generate_uid(entropy_srcs=[\"pepper\"])\n",
    "redacted_df[\"download_id\"] = [\n",
    "    hashlib.sha256(\n",
    "        (\n",
    "            row.study_instance_uid + \"_\" + row.series_instance_uid\n",
    "        ).encode()\n",
    "    ).hexdigest()\n",
    "    for row in redacted_df.itertuples(index=False)\n",
    "]\n",
    "redacted_df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And here we have it, this is our mapping DataFrame which references the folder where the files\n",
    "were downloaded, their de-identified IDs and their study and series instance UID.\n",
    "\n",
    "In our system, we have a specific URL that can be used to obtain de-identified studies. The\n",
    "`deidentified_study_instance_uid` and `deidentified_series_instance_uid` are the IDs that can be\n",
    "found in the newly downloaded DICOM files. If your system does not de-identify the studies, these\n",
    " two IDs will simply be the original IDs of the DICOM file."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}