examples/notebooks/intro/dpk_intro_1_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866",
   "metadata": {
    "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866"
   },
   "source": [
    "# Data Prep Kit Demo 1 - Python version\n",
    "\n",
    "This notebook will introduce DPK and showcase some of it's capabilities.\n",
    "\n",
    "Here is the workflow\n",
    "\n",
    "![](https://raw.githubusercontent.com/IBM/data-prep-kit/dev/examples/notebooks/intro/images/data-prep-kit-3-workflow.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15976e3",
   "metadata": {
    "id": "b15976e3"
   },
   "source": [
    "## How to run this notebook\n",
    "\n",
    "Two options:\n",
    "\n",
    "- **Option 1 - Google Colab:** easiest option.  no setup required.  Click this link to open this on google colab.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/IBM/data-prep-kit/blob/dev/examples/notebooks/intro/dpk_intro_1_python.ipynb)\n",
    "- **Option 2 - Local python dev environment:**  Setup using this [guide](../../../README.md#-getting-started)\n",
    "\n",
    "The notebook will work as in both environments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb8b0d5c",
   "metadata": {
    "id": "eb8b0d5c"
   },
   "source": [
    "## Step-1: Inspect the Data\n",
    "\n",
    "We will use simple PDFs about Solar system.  The files are [here](https://github.com/IBM/data-prep-kit/tree/dev/examples/notebooks/intro/input/solar-system)\n",
    "\n",
    "- [earth.pdf](https://github.com/IBM/data-prep-kit/blob/dev/examples/notebooks/intro/input/solar-system/earth.pdf)\n",
    "- [mars.pdf](https://github.com/IBM/data-prep-kit/blob/dev/examples/notebooks/intro/input/solar-system/mars.pdf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a0ab6e",
   "metadata": {
    "id": "39a0ab6e"
   },
   "source": [
    "## Step-2: Figure out Runtime Environment\n",
    "\n",
    "### 2.1 - Determine runtime\n",
    "\n",
    "Determine if we are running on Google colab or local python environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1fe354b7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1fe354b7",
    "outputId": "5c153f72-08ed-4d6e-ccc7-dae851e7fd8b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NOT in Colab\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if os.getenv(\"COLAB_RELEASE_TAG\"):\n",
    "   print(\"Running in Colab\")\n",
    "   RUNNING_IN_COLAB = True\n",
    "else:\n",
    "   print(\"NOT in Colab\")\n",
    "   RUNNING_IN_COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7c104b",
   "metadata": {
    "id": "8e7c104b"
   },
   "source": [
    "### 2.2 -Download Data if running on Google Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3309799e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3309799e",
    "outputId": "99530315-6dd5-405d-dbde-61e2332e441b"
   },
   "outputs": [],
   "source": [
    "if RUNNING_IN_COLAB:\n",
    "    !mkdir -p 'input/solar-system'\n",
    "    !wget -O 'input/solar-system/earth.pdf'  'https://raw.githubusercontent.com/IBM/data-prep-kit/dev/examples/notebooks/intro/input/solar-system/earth.pdf'\n",
    "    !wget -O 'input/solar-system/mars.pdf'  'https://raw.githubusercontent.com/IBM/data-prep-kit/dev/examples/notebooks/intro/input/solar-system/mars.pdf'\n",
    "    !wget -O 'my_utils.py'  'https://raw.githubusercontent.com/IBM/data-prep-kit/dev/examples/notebooks/intro/my_utils.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5dc2b68",
   "metadata": {
    "id": "a5dc2b68"
   },
   "source": [
    "### 2.3 - Install dependencies if running on Google Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1fcec577",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "1fcec577",
    "outputId": "0f77fc39-ffeb-48da-ce6f-1750d8d3ad62"
   },
   "outputs": [],
   "source": [
    "if RUNNING_IN_COLAB:\n",
    "    ! pip install  --default-timeout=100  \\\n",
    "        data-prep-toolkit==0.2.1 \\\n",
    "        data-prep-toolkit-transforms==0.2.1 \\\n",
    "        deepsearch-toolkit\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "243322b8",
   "metadata": {
    "id": "243322b8"
   },
   "source": [
    "### 2.4 - Restart Runtime\n",
    "\n",
    "After installing dependencies, be sure <font color=\"red\">restart runtime</font>, so libraries will be loaded\n",
    "\n",
    "You do this by going to **`Runtime --> Restart Session`**\n",
    "\n",
    "Then you can continue to the next step (no need to re-run the notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b10be1",
   "metadata": {
    "id": "e8b10be1"
   },
   "source": [
    "## Step-2: Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356c66f7",
   "metadata": {
    "id": "356c66f7"
   },
   "source": [
    "### 2.1 - Basic Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e4YMZrBuFycl",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e4YMZrBuFycl",
    "outputId": "d7ee9449-4f21-4c9a-fa54-14b7f28d764a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NOT in Colab\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if os.getenv(\"COLAB_RELEASE_TAG\"):\n",
    "   print(\"Running in Colab\")\n",
    "   RUNNING_IN_COLAB = True\n",
    "else:\n",
    "   print(\"NOT in Colab\")\n",
    "   RUNNING_IN_COLAB = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "33345487",
   "metadata": {
    "id": "33345487"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "## Configuration\n",
    "class MyConfig:\n",
    "    pass\n",
    "\n",
    "MY_CONFIG = MyConfig ()\n",
    "\n",
    "MY_CONFIG.INPUT_DATA_DIR = 'input/solar-system'\n",
    "\n",
    "MY_CONFIG.OUTPUT_FOLDER = \"output\"\n",
    "MY_CONFIG.OUTPUT_FOLDER_FINAL = os.path.join(MY_CONFIG.OUTPUT_FOLDER , \"output_final\")\n",
    "\n",
    "## Embedding model\n",
    "MY_CONFIG.EMBEDDING_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b15e6827",
   "metadata": {
    "id": "b15e6827"
   },
   "outputs": [],
   "source": [
    "## Add parent dir to path\n",
    "import os,sys\n",
    "\n",
    "this_dir = os.path.abspath('')\n",
    "parent_dir = os.path.dirname(this_dir)\n",
    "sys.path.append (os.path.abspath (parent_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72510ae6-48b0-4b88-9e13-a623281c3a63",
   "metadata": {
    "id": "72510ae6-48b0-4b88-9e13-a623281c3a63"
   },
   "source": [
    "### 2.2 - Setup input/outpur directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "60ac8bee-0960-4309-b225-d7a211b14262",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "60ac8bee-0960-4309-b225-d7a211b14262",
    "outputId": "4d5511fb-1c6f-47df-e5ea-2c1b354d262f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Cleared output directory\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "import shutil\n",
    "\n",
    "if not os.path.exists(MY_CONFIG.INPUT_DATA_DIR ):\n",
    "    raise Exception (f\"❌ Input folder MY_CONFIG.INPUT_DATA_DIR = '{MY_CONFIG.INPUT_DATA_DIR}' not found\")\n",
    "\n",
    "output_parquet_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '01_parquet_out')\n",
    "output_chunk_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '02_chunk_out')\n",
    "output_docid_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '03_docid_out')\n",
    "output_exact_dedupe_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '04_exact_dedupe_out')\n",
    "output_embeddings_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '05_embeddings_out')\n",
    "\n",
    "## clear output folder\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER, ignore_errors=True)\n",
    "shutil.os.makedirs(MY_CONFIG.OUTPUT_FOLDER, exist_ok=True)\n",
    "\n",
    "print (\"✅ Cleared output directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb",
   "metadata": {
    "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb"
   },
   "source": [
    "## Step-3: pdf2parquet -  Convert data from PDF to Parquet\n",
    "\n",
    "This step is reading the input folder containing all PDF files and ingest them in a parquet table using the [Docling package](https://github.com/DS4SD/docling).\n",
    "The documents are converted into a JSON format which allows to easily chunk it in the later steps.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c574c4-9dc4-4dab-9ad6-b5338207e67a",
   "metadata": {
    "id": "c0c574c4-9dc4-4dab-9ad6-b5338207e67a"
   },
   "source": [
    "### 3.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "482605b2-d814-456d-9195-49a2ec454ef0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "482605b2-d814-456d-9195-49a2ec454ef0",
    "outputId": "c50847d4-f2c7-4559-f5f7-d6a3d025027d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-1: Processing input='input/solar-system' --> output='output/01_parquet_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE = 1\n",
    "\n",
    "input_folder = MY_CONFIG.INPUT_DATA_DIR\n",
    "output_folder =  output_parquet_dir\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b",
   "metadata": {
    "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b"
   },
   "source": [
    "### 3.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b0cd8ebd-bf71-42d6-a397-8df0c7b66a26",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 657,
     "referenced_widgets": [
      "97b603697cfa4b4ea4e6735b6768ca35",
      "e87e8d3262c54cfaaa8768505edacda3",
      "b78aa40816e44f7fbebcb24ca68818b3",
      "7053c9606a414e978636a7e241909504",
      "da0787b239764847a731083997780a85",
      "553f3c16839a49d79591d0fc4862bed6",
      "c0eb5bc8f6ee427ca42204b3c56f9a4e",
      "9d184ed175f0403fb03c2e13dfd04e0a",
      "724778729161445c98b187031ae4f67c",
      "1cb3bbf7d724411cbe9831543a4aecc0",
      "06f9b33494984e4885d5aad813d1d2bc"
     ]
    },
    "id": "b0cd8ebd-bf71-42d6-a397-8df0c7b66a26",
    "outputId": "01d207fb-983d-40b2-e5f6-e38e3789110a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:39 INFO - pdf2parquet parameters are : {'artifacts_path': None, 'contents_type': <pdf2parquet_contents_types.JSON: 'application/json'>, 'do_table_structure': True, 'do_ocr': True, 'double_precision': 8}\n",
      "13:34:39 INFO - pipeline id pipeline_id\n",
      "13:34:39 INFO - code location None\n",
      "13:34:39 INFO - data factory data_ is using local data access: input_folder - input/solar-system output_folder - output/01_parquet_out\n",
      "13:34:39 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:34:39 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
      "13:34:39 INFO - orchestrator pdf2parquet started at 2024-10-18 13:34:39\n",
      "13:34:39 INFO - Number of files is 2, source profile {'max_file_size': 0.055823326110839844, 'min_file_size': 0.0551910400390625, 'total_file_size': 0.11101436614990234}\n",
      "13:34:39 INFO - Initializing models\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "750f3b6951094b2eb68490c7f5f98148",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:43 INFO - Completed 1 files (50.0%) in 0.013 min\n",
      "13:34:44 INFO - Completed 2 files (100.0%) in 0.027 min\n",
      "13:34:44 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:34:44 INFO - done flushing in 0.0 sec\n",
      "13:34:44 INFO - Completed execution in 0.084 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:1 completed successfully\n",
      "CPU times: user 11.2 s, sys: 1.56 s, total: 12.7 s\n",
      "Wall time: 7.75 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import ast\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from pdf2parquet_transform import (\n",
    "    pdf2parquet_contents_type_cli_param,\n",
    "    pdf2parquet_contents_types,\n",
    ")\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from pdf2parquet_transform_python import Pdf2ParquetPythonTransformConfiguration\n",
    "\n",
    "from data_processing.utils import GB, ParamsUtils\n",
    "\n",
    "\n",
    "# create parameters\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "ingest_config = {\n",
    "    pdf2parquet_contents_type_cli_param: pdf2parquet_contents_types.JSON,\n",
    "}\n",
    "\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    \"data_files_to_use\": ast.literal_eval(\"['.pdf']\"),\n",
    "}\n",
    "\n",
    "\n",
    "sys.argv = ParamsUtils.dict_to_req(d=(params | ingest_config))\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(Pdf2ParquetPythonTransformConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca790e0",
   "metadata": {
    "id": "5ca790e0"
   },
   "source": [
    "### 3.3 - Inspect Generated output\n",
    "\n",
    "Here we should see one entry per input file processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fe59563d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "id": "fe59563d",
    "outputId": "346e0584-bdde-4705-8c2a-f3c1582cd7e7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output dimensions (rows x columns)=  (2, 12)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_id</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename                                           contents  num_pages  \\\n",
       "0   mars.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...          1   \n",
       "1  earth.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...          1   \n",
       "\n",
       "   num_tables  num_doc_elements                           document_id  ext  \\\n",
       "0           0                11  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2  pdf   \n",
       "1           0                11  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c  pdf   \n",
       "\n",
       "                                                hash  size  \\\n",
       "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "1  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \n",
       "0  2024-10-18T13:34:44.259545          0.845978        mars.pdf  \n",
       "1  2024-10-18T13:34:43.410297          0.794765       earth.pdf  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from my_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Output dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(5)\n",
    "\n",
    "## To display certain columns\n",
    "#parquet_df[['column1', 'column2', 'column3']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5058a21",
   "metadata": {
    "id": "e5058a21"
   },
   "source": [
    "\n",
    "### 3.4 - Understand the output\n",
    "\n",
    "Here are some interesting attributes to note:\n",
    "\n",
    "- **filename** : original filename\n",
    "- **contents** : text\n",
    "- **document_id**: unique id (UUID) assignd to this document\n",
    "- **hash** : hash of document\n",
    "- **pdf_convert_time** : time to convert this pdf in seconds\n",
    "\n",
    "Let's inspect the **contents** column.  See how the text is being divided up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f870e624",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f870e624",
    "outputId": "0b4c054f-3a8a-4db3-f32f-17bd1466b102"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'_name': '',\n",
      " 'description': {'logs': []},\n",
      " 'equations': [],\n",
      " 'figures': [],\n",
      " 'file-info': {'#-pages': 1,\n",
      "               'document-hash': '1a83f43f3a202e3f203c1263e36961ecc45d401aad488f638fc5559a584333b2',\n",
      "               'filename': 'mars.pdf',\n",
      "               'page-hashes': [{'hash': '551fe7a9bde2a9302f150c0a79a13fcc0868fcf73ac6afb80be645c1174734a0',\n",
      "                                'model': 'default',\n",
      "                                'page': 1}]},\n",
      " 'footnotes': [],\n",
      " 'main-text': [{'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.35137939,\n",
      "                                   654.45184326,\n",
      "                                   169.88169861,\n",
      "                                   667.98492432],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 4]}],\n",
      "                'text': 'Mars',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.09541321,\n",
      "                                   630.68127441,\n",
      "                                   210.66503906,\n",
      "                                   642.34405518],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 12]}],\n",
      "                'text': 'Solar System',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [132.84518433,\n",
      "                                   588.96014404,\n",
      "                                   479.40917969,\n",
      "                                   623.02520752],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 205]}],\n",
      "                'text': 'Our solar system is a vast and fascinating expanse, '\n",
      "                        'comprising eight planets, five dwarf planets, '\n",
      "                        'numerous moons, asteroids, comets, and other '\n",
      "                        'celestial bodies. At its center lies the star we call '\n",
      "                        'the Sun.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [133.18510437,\n",
      "                                   570.83258057,\n",
      "                                   374.99838257,\n",
      "                                   581.07043457],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 54]}],\n",
      "                'text': 'For more details about the Solar system see Chapter '\n",
      "                        '1.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.22866821,\n",
      "                                   542.98168945,\n",
      "                                   163.86282349,\n",
      "                                   554.45288086],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 4]}],\n",
      "                'text': 'Mars',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [132.87440491,\n",
      "                                   500.84011841,\n",
      "                                   477.48345947,\n",
      "                                   534.55810547],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 196]}],\n",
      "                'text': 'Mars, the fourth planet from the Sun, is a cold, '\n",
      "                        'desert world with a thin atmosphere composed '\n",
      "                        'primarily of carbon dioxide. Its reddish hue comes '\n",
      "                        'from iron oxide, or rust, prevalent on its surface.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.2026062,\n",
      "                                   482.90710449,\n",
      "                                   237.04431152,\n",
      "                                   493.07443237],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 23]}],\n",
      "                'text': 'Basic facts about Mars:',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   453.019104,\n",
      "                                   477.48171997,\n",
      "                                   474.9703064],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 78]}],\n",
      "                'text': '· Distance from the Sun: Average of 228 million '\n",
      "                        'kilometers (142 million miles)',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   440.79351807,\n",
      "                                   431.73287964,\n",
      "                                   451.2142334],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 64]}],\n",
      "                'text': '· Rotation Period: 24.6 hours (one Martian day - '\n",
      "                        'called a \"sol\")',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   429.10913086,\n",
      "                                   365.9559021,\n",
      "                                   438.83737183],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 44]}],\n",
      "                'text': '· Moons: Two small moons, Phobos and Deimos.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Page-footer',\n",
      "                'prov': [{'bbox': [303.13299561,\n",
      "                                   87.20314026,\n",
      "                                   308.11428833,\n",
      "                                   96.51646423],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 1]}],\n",
      "                'text': '1',\n",
      "                'type': 'page-footer'}],\n",
      " 'page-dimensions': [{'height': 792.0, 'page': 1, 'width': 612.0}],\n",
      " 'page-footers': [],\n",
      " 'page-headers': [],\n",
      " 'tables': [],\n",
      " 'type': 'pdf-document'}\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "import json\n",
    "\n",
    "pprint.pprint (json.loads(output_df.iloc[0, ]['contents']))\n",
    "# json.loads(output_df.iloc[0, ]['contents'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e1a10c2d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e1a10c2d",
    "outputId": "c1d992c2-faa8-40cd-c375-857970201daa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'_name': '',\n",
      " 'description': {'logs': []},\n",
      " 'equations': [],\n",
      " 'figures': [],\n",
      " 'file-info': {'#-pages': 1,\n",
      "               'document-hash': '7401ae81637dbb89e7040dcd5945bbfb75ff8648bb761c69f8a1595e86538748',\n",
      "               'filename': 'earth.pdf',\n",
      "               'page-hashes': [{'hash': 'ca802e4bd5a3301792808caea2a47db51f0520888875b77fc230c99ee851c19b',\n",
      "                                'model': 'default',\n",
      "                                'page': 1}]},\n",
      " 'footnotes': [],\n",
      " 'main-text': [{'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.30961609,\n",
      "                                   654.45184326,\n",
      "                                   174.04208374,\n",
      "                                   667.93347168],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 5]}],\n",
      "                'text': 'Earth',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.12528992,\n",
      "                                   630.69073486,\n",
      "                                   210.66503906,\n",
      "                                   642.27935791],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 12]}],\n",
      "                'text': 'Solar System',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [132.87112427,\n",
      "                                   588.96014404,\n",
      "                                   479.40917969,\n",
      "                                   623.04595947],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 205]}],\n",
      "                'text': 'Our solar system is a vast and fascinating expanse, '\n",
      "                        'comprising eight planets, five dwarf planets, '\n",
      "                        'numerous moons, asteroids, comets, and other '\n",
      "                        'celestial bodies. At its center lies the star we call '\n",
      "                        'the Sun.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [133.20942688,\n",
      "                                   570.81555176,\n",
      "                                   375.57919312,\n",
      "                                   581.08459473],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 54]}],\n",
      "                'text': 'For more details about our Solar system see Chapter '\n",
      "                        '1.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Section-header',\n",
      "                'prov': [{'bbox': [133.15542603,\n",
      "                                   542.98168945,\n",
      "                                   167.32983398,\n",
      "                                   554.36669922],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 5]}],\n",
      "                'text': 'Earth',\n",
      "                'type': 'subtitle-level-1'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [132.91053772,\n",
      "                                   512.46295166,\n",
      "                                   477.84887695,\n",
      "                                   534.48431396],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 107]}],\n",
      "                'text': \"Earth is the third planet from the Sun. It's our home \"\n",
      "                        'planet. Earth is the only place we know of with life.',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Text',\n",
      "                'prov': [{'bbox': [133.30151367,\n",
      "                                   494.86206055,\n",
      "                                   240.17156982,\n",
      "                                   505.07229614],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 24]}],\n",
      "                'text': 'Basic facts about Earth:',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   464.97409058,\n",
      "                                   477.47979736,\n",
      "                                   487.02810669],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 79]}],\n",
      "                'text': '· Distance from the Sun: Average of 149.6 million '\n",
      "                        'kilometers (93 million miles)',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   452.86901855,\n",
      "                                   317.90722656,\n",
      "                                   463.24041748],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 37]}],\n",
      "                'text': '· Rotation Period: 24 hours (one day)',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'List-item',\n",
      "                'prov': [{'bbox': [145.94500732,\n",
      "                                   440.71496582,\n",
      "                                   396.66357422,\n",
      "                                   451.19915771],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 52]}],\n",
      "                'text': '· Moons: One moon, called Luna or simply \"the Moon\".',\n",
      "                'type': 'paragraph'},\n",
      "               {'name': 'Page-footer',\n",
      "                'prov': [{'bbox': [303.13299561,\n",
      "                                   87.20314026,\n",
      "                                   308.11428833,\n",
      "                                   96.53633118],\n",
      "                          'page': 1,\n",
      "                          'span': [0, 1]}],\n",
      "                'text': '1',\n",
      "                'type': 'page-footer'}],\n",
      " 'page-dimensions': [{'height': 792.0, 'page': 1, 'width': 612.0}],\n",
      " 'page-footers': [],\n",
      " 'page-headers': [],\n",
      " 'tables': [],\n",
      " 'type': 'pdf-document'}\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint (json.loads(output_df.iloc[1, ]['contents']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72274586",
   "metadata": {
    "id": "72274586"
   },
   "source": [
    "##  Step-4: Doc chunks\n",
    "\n",
    "In the previous step, we have extracted text from oru PDFs.  But we have the content of entire file as 'one row' in our parquet output.\n",
    "\n",
    "In this step, we are going to split the documents in chunks, according to their layout segmentation.\n",
    "\n",
    "This transform uses [Quackling](https://github.com/DS4SD/quackling) `HierarchicalChunker`\n",
    "to chunk according to the document layout segmentation, i.e. respecting the original document components as paragraphs, tables, enumerations, etc.\n",
    "It relies on documents converted with the Docling library in the [pdf2parquet transform](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/pdf2parquet/python/README.md) using the option `contents_type: \"application/json\"`,\n",
    "which provides the required JSON structure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96198fa6",
   "metadata": {
    "id": "96198fa6"
   },
   "source": [
    "### 4.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "305f00a3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "305f00a3",
    "outputId": "dd511f34-bab3-4dde-d938-493debb02e5e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-2: Processing input='output/01_parquet_out' --> output='output/02_chunk_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE = 2\n",
    "\n",
    "input_folder = output_parquet_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_chunk_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369f2cd1",
   "metadata": {
    "id": "369f2cd1"
   },
   "source": [
    "### 4.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5b7b18d5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5b7b18d5",
    "outputId": "e0b87171-9d66-473f-e66a-e4b6ae3c3f66"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:45 INFO - doc_chunk parameters are : {'chunking_type': <chunking_types.DL_JSON: 'dl_json'>, 'content_column_name': 'contents', 'doc_id_column_name': 'document_id', 'dl_min_chunk_len': None, 'output_chunk_column_name': 'contents', 'output_source_doc_id_column_name': 'source_document_id', 'output_jsonpath_column_name': 'doc_jsonpath', 'output_pageno_column_name': 'page_number', 'output_bbox_column_name': 'bbox'}\n",
      "13:34:45 INFO - pipeline id pipeline_id\n",
      "13:34:45 INFO - code location None\n",
      "13:34:45 INFO - data factory data_ is using local data access: input_folder - output/01_parquet_out output_folder - output/02_chunk_out\n",
      "13:34:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:34:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:34:45 INFO - orchestrator doc_chunk started at 2024-10-18 13:34:45\n",
      "13:34:45 INFO - Number of files is 2, source profile {'max_file_size': 0.02239513397216797, 'min_file_size': 0.02167987823486328, 'total_file_size': 0.04407501220703125}\n",
      "13:34:45 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "13:34:45 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "13:34:45 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:34:45 INFO - done flushing in 0.0 sec\n",
      "13:34:45 INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:2 completed successfully\n",
      "CPU times: user 826 ms, sys: 101 ms, total: 928 ms\n",
      "Wall time: 923 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from doc_chunk_transform_python import DocChunkPythonTransformConfiguration\n",
    "\n",
    "\n",
    "# Prepare the commandline params\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # doc_chunk arguments\n",
    "    # ...\n",
    "}\n",
    "\n",
    "# Pass the commandline params\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(DocChunkPythonTransformConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213afdf6",
   "metadata": {
    "id": "213afdf6"
   },
   "source": [
    "### 4.3 - Inspect Generated output\n",
    "\n",
    "We would see documents are split into many chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8138d43",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 897
    },
    "id": "d8138d43",
    "outputId": "fd01e0cb-899e-4c73-d50e-5f4e6f5ff802"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files processed : 2\n",
      "Chunks created : 8\n",
      "Input data dimensions (rows x columns)=  (2, 12)\n",
      "Output data dimensions (rows x columns)=  (8, 16)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.84518433, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "0   mars.pdf          1           0                11  pdf   \n",
       "1   mars.pdf          1           0                11  pdf   \n",
       "2   mars.pdf          1           0                11  pdf   \n",
       "3   mars.pdf          1           0                11  pdf   \n",
       "4  earth.pdf          1           0                11  pdf   \n",
       "5  earth.pdf          1           0                11  pdf   \n",
       "6  earth.pdf          1           0                11  pdf   \n",
       "7  earth.pdf          1           0                11  pdf   \n",
       "\n",
       "                                                hash  size  \\\n",
       "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "3  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "7  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \\\n",
       "0  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "1  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "2  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "3  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "4  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "5  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "6  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "7  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "\n",
       "                     source_document_id  \\\n",
       "0  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "1  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "2  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "3  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "4  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "5  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "6  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "7  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "\n",
       "                                            contents    doc_jsonpath  \\\n",
       "0  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "1  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
       "2  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
       "3  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
       "4  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "5  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
       "6  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
       "7  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
       "\n",
       "   page_number                                               bbox  \\\n",
       "0            1  [132.84518433, 588.96014404, 479.40917969, 623...   \n",
       "1            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
       "2            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
       "3            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
       "4            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
       "5            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
       "6            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
       "7            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
       "\n",
       "                                         document_id  \n",
       "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...  \n",
       "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...  \n",
       "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...  \n",
       "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...  \n",
       "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...  \n",
       "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...  \n",
       "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...  \n",
       "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from my_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (f\"Files processed : {input_df.shape[0]:,}\")\n",
    "print (f\"Chunks created : {output_df.shape[0]:,}\")\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e9ca75c",
   "metadata": {
    "id": "9e9ca75c"
   },
   "source": [
    "### 4.4 - Understanding the Output\n",
    "\n",
    "Here we see 2 PDF files are split into 6 chunks.  Basically we see the documents are being split along 'natural boundaris' - paragraphs and bullet points\n",
    "\n",
    "See how **document_id** is carried throughout.  This helps us identify original documents.\n",
    "\n",
    "Also note **contents** is now plain text (not JSON as before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3090c950",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "3090c950",
    "outputId": "0f4b6771-8d38-4a27-c756-21f916b23a4f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename                                           contents\n",
       "0   mars.pdf  Solar System\\nOur solar system is a vast and f...\n",
       "1   mars.pdf  Solar System\\nFor more details about the Solar...\n",
       "2   mars.pdf  Mars\\nMars, the fourth planet from the Sun, is...\n",
       "3   mars.pdf  Basic facts about Mars:\\n· Distance from the S...\n",
       "4  earth.pdf  Solar System\\nOur solar system is a vast and f...\n",
       "5  earth.pdf  Solar System\\nFor more details about our Solar...\n",
       "6  earth.pdf  Earth\\nEarth is the third planet from the Sun....\n",
       "7  earth.pdf  Earth\\nBasic facts about Earth:\\n· Distance fr..."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_df[['filename', 'contents']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d5f151ae",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d5f151ae",
    "outputId": "a4c491b2-53db-4d71-da24-4479de8d1d65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== mars.pdf ===========\n",
      "-------Chunk 0------\n",
      "Solar System\n",
      "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
      "-------\n",
      "-------Chunk 1------\n",
      "Solar System\n",
      "For more details about the Solar system see Chapter 1.\n",
      "-------\n",
      "-------Chunk 2------\n",
      "Mars\n",
      "Mars, the fourth planet from the Sun, is a cold, desert world with a thin atmosphere composed primarily of carbon dioxide. Its reddish hue comes from iron oxide, or rust, prevalent on its surface.\n",
      "-------\n",
      "-------Chunk 3------\n",
      "Basic facts about Mars:\n",
      "· Distance from the Sun: Average of 228 million kilometers (142 million miles)\n",
      "· Rotation Period: 24.6 hours (one Martian day - called a \"sol\")\n",
      "· Moons: Two small moons, Phobos and Deimos.\n",
      "-------\n",
      "========== earth.pdf ===========\n",
      "-------Chunk 0------\n",
      "Solar System\n",
      "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
      "-------\n",
      "-------Chunk 1------\n",
      "Solar System\n",
      "For more details about our Solar system see Chapter 1.\n",
      "-------\n",
      "-------Chunk 2------\n",
      "Earth\n",
      "Earth is the third planet from the Sun. It's our home planet. Earth is the only place we know of with life.\n",
      "-------\n",
      "-------Chunk 3------\n",
      "Earth\n",
      "Basic facts about Earth:\n",
      "· Distance from the Sun: Average of 149.6 million kilometers (93 million miles)\n",
      "· Rotation Period: 24 hours (one day)\n",
      "· Moons: One moon, called Luna or simply \"the Moon\".\n",
      "-------\n"
     ]
    }
   ],
   "source": [
    "for f in output_df['filename'].unique():\n",
    "    print ('==========' , f, '===========')\n",
    "    chunks = output_df[output_df['filename'] == f]['contents']\n",
    "    for idx , chunk in enumerate(chunks):\n",
    "        print (f'-------Chunk {idx}------\\n{chunk}\\n-------')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad1c60d",
   "metadata": {
    "id": "7ad1c60d"
   },
   "source": [
    "## Step-5:  DOC ID generation of Chunks\n",
    "\n",
    "This transform annotates documents with document \"ids\". It supports the following transformations of the original data:\n",
    "\n",
    " - Adding document hash: this enables the addition of a document hash-based id to the data. The hash is calculated with `hashlib.sha256(doc.encode(\"utf-8\")).hexdigest()`. To enable this annotation, set **hash_column** to the name of the column, where you want to store it.\n",
    " - Adding integer document id: this allows the addition of an integer document id to the data that is unique across all rows in all tables provided to the transform() method. To enable this annotation, set **int_id_column** to the name of the column, where you want to store it.\n",
    "\n",
    "**This is a pre-requisite for fuzzy dedup** in the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1afaa0fd",
   "metadata": {
    "id": "1afaa0fd"
   },
   "source": [
    "### 5.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6ffd6f54",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6ffd6f54",
    "outputId": "1784c80d-6309-4913-9f55-c018b978968f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-3: Processing input='output/02_chunk_out' --> output='output/03_docid_out'\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Input for this stage is the output of exact dedeup component\n",
    "# output of this component makes it possible for fdedup component to run on data.\n",
    "\n",
    "STAGE  = 3\n",
    "\n",
    "input_folder = output_chunk_dir\n",
    "output_folder =  output_docid_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f78a51b7",
   "metadata": {
    "id": "f78a51b7"
   },
   "source": [
    "### 5.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5fc77557",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5fc77557",
    "outputId": "db2b8670-543e-4073-9c7d-3f9ef5f4317e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:45 INFO - Doc id parameters are : {'doc_column': 'contents', 'hash_column': 'chunk_hash', 'int_column': 'chunk_id', 'start_id': 0}\n",
      "13:34:45 INFO - pipeline id pipeline_id\n",
      "13:34:45 INFO - code location None\n",
      "13:34:45 INFO - data factory data_ is using local data access: input_folder - output/02_chunk_out output_folder - output/03_docid_out\n",
      "13:34:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:34:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:34:45 INFO - orchestrator doc_id started at 2024-10-18 13:34:45\n",
      "13:34:45 INFO - Number of files is 2, source profile {'max_file_size': 0.008975982666015625, 'min_file_size': 0.008897781372070312, 'total_file_size': 0.017873764038085938}\n",
      "13:34:45 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "13:34:45 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "13:34:45 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:34:45 INFO - done flushing in 0.0 sec\n",
      "13:34:45 INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:3 completed successfully\n",
      "CPU times: user 12.8 ms, sys: 3.7 ms, total: 16.5 ms\n",
      "Wall time: 13.1 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from doc_id_transform_python import DocIDPythonTransformRuntimeConfiguration\n",
    "\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # orchestrator\n",
    "    # doc id configuration\n",
    "    \"doc_id_doc_column\": \"contents\",\n",
    "    \"doc_id_hash_column\": \"chunk_hash\",\n",
    "    \"doc_id_int_column\": \"chunk_id\",\n",
    "}\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# launch\n",
    "\n",
    "launcher = PythonTransformLauncher(DocIDPythonTransformRuntimeConfiguration())\n",
    "\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9a8c1fa",
   "metadata": {
    "id": "a9a8c1fa"
   },
   "source": [
    "### 5.3 - Inspect Generated output\n",
    "\n",
    "You will notice we have two extra columns\n",
    "\n",
    "- **hash_column**\n",
    "- **int_id_column**\n",
    "\n",
    "But still the same number or rows as before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "da9adede",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 860
    },
    "id": "da9adede",
    "outputId": "036db4ca-12f6-4b3e-9d7f-fa70e494870d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (8, 16)\n",
      "Output data dimensions (rows x columns)=  (8, 18)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.84518433, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "0   mars.pdf          1           0                11  pdf   \n",
       "1   mars.pdf          1           0                11  pdf   \n",
       "2   mars.pdf          1           0                11  pdf   \n",
       "3   mars.pdf          1           0                11  pdf   \n",
       "4  earth.pdf          1           0                11  pdf   \n",
       "5  earth.pdf          1           0                11  pdf   \n",
       "6  earth.pdf          1           0                11  pdf   \n",
       "7  earth.pdf          1           0                11  pdf   \n",
       "\n",
       "                                                hash  size  \\\n",
       "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "3  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "7  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \\\n",
       "0  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "1  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "2  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "3  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "4  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "5  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "6  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "7  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "\n",
       "                     source_document_id  \\\n",
       "0  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "1  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "2  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "3  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "4  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "5  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "6  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "7  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "\n",
       "                                            contents    doc_jsonpath  \\\n",
       "0  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "1  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
       "2  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
       "3  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
       "4  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "5  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
       "6  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
       "7  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
       "\n",
       "   page_number                                               bbox  \\\n",
       "0            1  [132.84518433, 588.96014404, 479.40917969, 623...   \n",
       "1            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
       "2            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
       "3            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
       "4            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
       "5            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
       "6            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
       "7            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
       "\n",
       "                                         document_id  \\\n",
       "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
       "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...   \n",
       "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...   \n",
       "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...   \n",
       "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
       "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...   \n",
       "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...   \n",
       "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...   \n",
       "\n",
       "                                          chunk_hash  chunk_id  \n",
       "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         4  \n",
       "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...         5  \n",
       "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6  \n",
       "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7  \n",
       "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0  \n",
       "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1  \n",
       "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2  \n",
       "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...         3  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from my_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4692975c-49ff-41ae-810e-0f5bc0bbdc53",
   "metadata": {
    "id": "4692975c-49ff-41ae-810e-0f5bc0bbdc53"
   },
   "source": [
    "## Step-6: Exact Dedup\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5acfd3a2-a236-4143-bcfc-15804f1da7fe",
   "metadata": {
    "id": "5acfd3a2-a236-4143-bcfc-15804f1da7fe"
   },
   "source": [
    "### 6.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "4c7a1b94",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4c7a1b94",
    "outputId": "2f6f05bc-f6fd-4d66-ea01-ed89cd5b80f3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-4: Processing input='output/03_docid_out' --> output='output/04_exact_dedupe_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE  = 4\n",
    "\n",
    "input_folder = output_docid_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_exact_dedupe_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3661cb37-39c7-4b09-a784-925bfa9eaf1e",
   "metadata": {
    "id": "3661cb37-39c7-4b09-a784-925bfa9eaf1e"
   },
   "source": [
    "### 6.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a624b2b2-faad-4325-ac7d-53a840f564ef",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a624b2b2-faad-4325-ac7d-53a840f564ef",
    "outputId": "74dc0b75-58b5-4c97-9965-91315e8a98a5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:45 INFO - exact dedup params are {'doc_column': 'contents', 'doc_id_column': 'chunk_hash', 'use_snapshot': False, 'snapshot_directory': None}\n",
      "13:34:45 INFO - pipeline id pipeline_id\n",
      "13:34:45 INFO - code location None\n",
      "13:34:45 INFO - data factory data_ is using local data access: input_folder - output/03_docid_out output_folder - output/04_exact_dedupe_out\n",
      "13:34:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:34:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:34:45 INFO - orchestrator ededup started at 2024-10-18 13:34:45\n",
      "13:34:45 INFO - Number of files is 2, source profile {'max_file_size': 0.010180473327636719, 'min_file_size': 0.010101318359375, 'total_file_size': 0.02028179168701172}\n",
      "13:34:45 INFO - Starting from the beginning\n",
      "13:34:45 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "13:34:45 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "13:34:45 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:34:45 INFO - done flushing in 0.0 sec\n",
      "13:34:45 INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:4 completed successfully\n",
      "CPU times: user 17.6 ms, sys: 997 μs, total: 18.6 ms\n",
      "Wall time: 15.2 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from ededup_transform_python import EdedupPythonTransformRuntimeConfiguration\n",
    "\n",
    "\n",
    "# Prepare the commandline params\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # ededup parameters\n",
    "    \"ededup_doc_column\": \"contents\",\n",
    "    \"ededup_doc_id_column\": \"chunk_hash\",\n",
    "}\n",
    "\n",
    "# Pass the commandline params\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(EdedupPythonTransformRuntimeConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf1c3c3",
   "metadata": {
    "id": "eaf1c3c3"
   },
   "source": [
    "### 6.3 - Inspect Generated output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d824ebf6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 815
    },
    "id": "d824ebf6",
    "outputId": "68f55770-c750-4607-a205-ba183603019d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (8, 18)\n",
      "Output data dimensions (rows x columns)=  (7, 19)\n",
      "Input chunks before exact dedupe : 8\n",
      "Output chunks after exact dedupe : 7\n",
      "Duplicate chunks removed :   1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "      <th>removed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>5</td>\n",
       "      <td>[44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>6</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>0</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>1</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>2</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>3</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "0   mars.pdf          1           0                11  pdf   \n",
       "1   mars.pdf          1           0                11  pdf   \n",
       "2   mars.pdf          1           0                11  pdf   \n",
       "3  earth.pdf          1           0                11  pdf   \n",
       "4  earth.pdf          1           0                11  pdf   \n",
       "5  earth.pdf          1           0                11  pdf   \n",
       "6  earth.pdf          1           0                11  pdf   \n",
       "\n",
       "                                                hash  size  \\\n",
       "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "3  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \\\n",
       "0  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "1  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "2  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "3  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "4  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "5  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "6  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "\n",
       "                     source_document_id  \\\n",
       "0  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "1  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "2  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "3  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "4  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "5  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "6  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "\n",
       "                                            contents    doc_jsonpath  \\\n",
       "0  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
       "1  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
       "2  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
       "3  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "4  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
       "5  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
       "6  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
       "\n",
       "   page_number                                               bbox  \\\n",
       "0            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
       "1            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
       "2            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
       "3            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
       "4            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
       "5            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
       "6            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
       "\n",
       "                                         document_id  \\\n",
       "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...   \n",
       "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...   \n",
       "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...   \n",
       "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
       "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...   \n",
       "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...   \n",
       "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...   \n",
       "\n",
       "                                          chunk_hash  chunk_id  \\\n",
       "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...         5   \n",
       "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6   \n",
       "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7   \n",
       "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0   \n",
       "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1   \n",
       "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2   \n",
       "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...         3   \n",
       "\n",
       "                                             removed  \n",
       "0  [44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...  \n",
       "1                                                 []  \n",
       "2                                                 []  \n",
       "3                                                 []  \n",
       "4                                                 []  \n",
       "5                                                 []  \n",
       "6                                                 []  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from my_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "print (f\"Input chunks before exact dedupe : {input_df.shape[0]:,}\")\n",
    "print (f\"Output chunks after exact dedupe : {output_df.shape[0]:,}\")\n",
    "print (\"Duplicate chunks removed :  \", (input_df.shape[0] - output_df.shape[0]))\n",
    "\n",
    "output_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "82cc9bb0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "82cc9bb0",
    "outputId": "46d9e91d-c470-4e3e-e5c8-508c534dbceb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename                                           contents\n",
       "0   mars.pdf  Solar System\\nFor more details about the Solar...\n",
       "1   mars.pdf  Mars\\nMars, the fourth planet from the Sun, is...\n",
       "2   mars.pdf  Basic facts about Mars:\\n· Distance from the S...\n",
       "3  earth.pdf  Solar System\\nOur solar system is a vast and f...\n",
       "4  earth.pdf  Solar System\\nFor more details about our Solar...\n",
       "5  earth.pdf  Earth\\nEarth is the third planet from the Sun....\n",
       "6  earth.pdf  Earth\\nBasic facts about Earth:\\n· Distance fr..."
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_df[['filename', 'contents']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cc61dffa",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cc61dffa",
    "outputId": "7fb26043-8538-48b6-80b7-16ceb818c1a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== mars.pdf ===========\n",
      "-------Chunk 0------\n",
      "Solar System\n",
      "For more details about the Solar system see Chapter 1.\n",
      "-------\n",
      "-------Chunk 1------\n",
      "Mars\n",
      "Mars, the fourth planet from the Sun, is a cold, desert world with a thin atmosphere composed primarily of carbon dioxide. Its reddish hue comes from iron oxide, or rust, prevalent on its surface.\n",
      "-------\n",
      "-------Chunk 2------\n",
      "Basic facts about Mars:\n",
      "· Distance from the Sun: Average of 228 million kilometers (142 million miles)\n",
      "· Rotation Period: 24.6 hours (one Martian day - called a \"sol\")\n",
      "· Moons: Two small moons, Phobos and Deimos.\n",
      "-------\n",
      "========== earth.pdf ===========\n",
      "-------Chunk 0------\n",
      "Solar System\n",
      "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
      "-------\n",
      "-------Chunk 1------\n",
      "Solar System\n",
      "For more details about our Solar system see Chapter 1.\n",
      "-------\n",
      "-------Chunk 2------\n",
      "Earth\n",
      "Earth is the third planet from the Sun. It's our home planet. Earth is the only place we know of with life.\n",
      "-------\n",
      "-------Chunk 3------\n",
      "Earth\n",
      "Basic facts about Earth:\n",
      "· Distance from the Sun: Average of 149.6 million kilometers (93 million miles)\n",
      "· Rotation Period: 24 hours (one day)\n",
      "· Moons: One moon, called Luna or simply \"the Moon\".\n",
      "-------\n"
     ]
    }
   ],
   "source": [
    "for f in output_df['filename'].unique():\n",
    "    print ('==========' , f, '===========')\n",
    "    chunks = output_df[output_df['filename'] == f]['contents']\n",
    "    for idx , chunk in enumerate(chunks):\n",
    "        print (f'-------Chunk {idx}------\\n{chunk}\\n-------')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "383f40ba",
   "metadata": {
    "id": "383f40ba"
   },
   "source": [
    "### 6.4 - Understanding the output\n",
    "\n",
    "Remember we had 8 chunks initially.  Now we have 7!  One duplicate chunk is removed.\n",
    "\n",
    "If you look at the PDF, the following common paragraph in `earth.pdf` and `mars.pdf`  is removed from one of the documents!  Pretty neat, eh!\n",
    "\n",
    "```text\n",
    "## Solar System\n",
    "\n",
    "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85309751-8556-41c6-ac32-84acc941bc8d",
   "metadata": {
    "id": "85309751-8556-41c6-ac32-84acc941bc8d"
   },
   "source": [
    "  ## Step-7: Fuzzy Dedup\n",
    "\n",
    "And fuzzy dedupe is only available in RAY version.  So we will skip it here\n",
    "\n",
    "See this file [dpk_intro_1_ray.ipynb](dpk_intro_1_ray.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5370950a-2a3a-4143-8218-f9b4808099ba",
   "metadata": {
    "id": "5370950a-2a3a-4143-8218-f9b4808099ba"
   },
   "source": [
    "## Step-8:   Text encoding\n",
    "\n",
    "Encode text for the vector storage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85aba685",
   "metadata": {
    "id": "85aba685"
   },
   "source": [
    "### 8.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "20a153fa-fd56-401e-86be-4f7617affcc8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "20a153fa-fd56-401e-86be-4f7617affcc8",
    "outputId": "41d268f5-7cc6-432e-d56e-2ba882fbdba6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-6: Processing input='output/04_exact_dedupe_out' --> output='output/05_embeddings_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE  = 6\n",
    "\n",
    "input_folder = output_exact_dedupe_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_embeddings_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c97545f4",
   "metadata": {
    "id": "c97545f4"
   },
   "source": [
    "### 8.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "228df6b2-bc62-494b-9697-03ece98d7853",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "228df6b2-bc62-494b-9697-03ece98d7853",
    "outputId": "b2119b07-0654-45cd-f729-1396e18b24b1"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13:34:45 INFO - text_encoder parameters are : {'content_column_name': 'contents', 'output_embeddings_column_name': 'embeddings', 'model_name': 'sentence-transformers/all-MiniLM-L6-v2'}\n",
      "13:34:45 INFO - pipeline id pipeline_id\n",
      "13:34:45 INFO - code location None\n",
      "13:34:45 INFO - data factory data_ is using local data access: input_folder - output/04_exact_dedupe_out output_folder - output/05_embeddings_out\n",
      "13:34:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "13:34:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "13:34:45 INFO - orchestrator text_encoder started at 2024-10-18 13:34:45\n",
      "13:34:45 INFO - Number of files is 2, source profile {'max_file_size': 0.010450363159179688, 'min_file_size': 0.010318756103515625, 'total_file_size': 0.020769119262695312}\n",
      "13:34:47 INFO - Completed 1 files (50.0%) in 0.004 min\n",
      "13:34:47 INFO - Completed 2 files (100.0%) in 0.005 min\n",
      "13:34:47 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "13:34:47 INFO - done flushing in 0.0 sec\n",
      "13:34:47 INFO - Completed execution in 0.034 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:6 completed successfully\n",
      "CPU times: user 615 ms, sys: 146 ms, total: 761 ms\n",
      "Wall time: 2.24 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from text_encoder_local_python import TextEncoderPythonTransformConfiguration\n",
    "\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # text_encoder\n",
    "    \"text_encoder_model_name\": MY_CONFIG.EMBEDDING_MODEL,\n",
    "}\n",
    "\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(TextEncoderPythonTransformConfiguration())\n",
    "\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b734852c",
   "metadata": {
    "id": "b734852c"
   },
   "source": [
    "### 8.3 - Inspect Generated output\n",
    "\n",
    "You will see a column called `embeddings` added at the end.  This the text content converted into vectors or embeddings.  We used the model `sentence-transformers/all-MiniLM-L6-v2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7b1c1d09",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 760
    },
    "id": "7b1c1d09",
    "outputId": "018daa18-e5db-4483-d8d5-30aded80d5e3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (7, 19)\n",
      "Output data dimensions (rows x columns)=  (7, 20)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "      <th>removed</th>\n",
       "      <th>embeddings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Solar System\\nFor more details about the Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
       "      <td>5</td>\n",
       "      <td>[44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...</td>\n",
       "      <td>[-0.051861435, 0.0035226212, 0.030617002, 0.04...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
       "      <td>6</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.07728295, 0.024970993, -0.043180738, 0.0580...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
       "      <td>2800</td>\n",
       "      <td>2024-10-18T13:34:44.259545</td>\n",
       "      <td>0.845978</td>\n",
       "      <td>mars.pdf</td>\n",
       "      <td>6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2</td>\n",
       "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
       "      <td>7</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.10598018, 0.025460618, 0.023627337, 0.03905...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
       "      <td>$.main-text[2]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
       "      <td>0</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.0077404436, -0.02055944, 0.026426593, 0.011...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Solar System\\nFor more details about our Solar...</td>\n",
       "      <td>$.main-text[3]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
       "      <td>1</td>\n",
       "      <td>[]</td>\n",
       "      <td>[-0.062105548, -0.0053322907, 0.031277698, 0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
       "      <td>$.main-text[5]</td>\n",
       "      <td>1</td>\n",
       "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
       "      <td>2</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.072435796, -0.058001805, -0.019771898, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>pdf</td>\n",
       "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
       "      <td>2686</td>\n",
       "      <td>2024-10-18T13:34:43.410297</td>\n",
       "      <td>0.794765</td>\n",
       "      <td>earth.pdf</td>\n",
       "      <td>efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c</td>\n",
       "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
       "      <td>$.main-text[6]</td>\n",
       "      <td>1</td>\n",
       "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
       "      <td>3</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.091821924, 0.015197902, 0.07716932, 0.01711...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "0   mars.pdf          1           0                11  pdf   \n",
       "1   mars.pdf          1           0                11  pdf   \n",
       "2   mars.pdf          1           0                11  pdf   \n",
       "3  earth.pdf          1           0                11  pdf   \n",
       "4  earth.pdf          1           0                11  pdf   \n",
       "5  earth.pdf          1           0                11  pdf   \n",
       "6  earth.pdf          1           0                11  pdf   \n",
       "\n",
       "                                                hash  size  \\\n",
       "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
       "3  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \\\n",
       "0  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "1  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "2  2024-10-18T13:34:44.259545          0.845978        mars.pdf   \n",
       "3  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "4  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "5  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "6  2024-10-18T13:34:43.410297          0.794765       earth.pdf   \n",
       "\n",
       "                     source_document_id  \\\n",
       "0  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "1  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "2  6e9fd08a-a4e2-47da-b5a9-bb1e1a3ab6e2   \n",
       "3  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "4  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "5  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "6  efbdbcb9-f0af-42f0-b191-2f14ce3ddc7c   \n",
       "\n",
       "                                            contents    doc_jsonpath  \\\n",
       "0  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
       "1  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
       "2  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
       "3  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
       "4  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
       "5  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
       "6  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
       "\n",
       "   page_number                                               bbox  \\\n",
       "0            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
       "1            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
       "2            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
       "3            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
       "4            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
       "5            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
       "6            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
       "\n",
       "                                         document_id  \\\n",
       "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...   \n",
       "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...   \n",
       "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...   \n",
       "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
       "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...   \n",
       "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...   \n",
       "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...   \n",
       "\n",
       "                                          chunk_hash  chunk_id  \\\n",
       "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...         5   \n",
       "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6   \n",
       "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7   \n",
       "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0   \n",
       "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1   \n",
       "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2   \n",
       "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...         3   \n",
       "\n",
       "                                             removed  \\\n",
       "0  [44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...   \n",
       "1                                                 []   \n",
       "2                                                 []   \n",
       "3                                                 []   \n",
       "4                                                 []   \n",
       "5                                                 []   \n",
       "6                                                 []   \n",
       "\n",
       "                                          embeddings  \n",
       "0  [-0.051861435, 0.0035226212, 0.030617002, 0.04...  \n",
       "1  [0.07728295, 0.024970993, -0.043180738, 0.0580...  \n",
       "2  [0.10598018, 0.025460618, 0.023627337, 0.03905...  \n",
       "3  [0.0077404436, -0.02055944, 0.026426593, 0.011...  \n",
       "4  [-0.062105548, -0.0053322907, 0.031277698, 0.0...  \n",
       "5  [0.072435796, -0.058001805, -0.019771898, -0.0...  \n",
       "6  [0.091821924, 0.015197902, 0.07716932, 0.01711...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from my_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e12630-be6b-4188-a925-77117155617b",
   "metadata": {
    "id": "f5e12630-be6b-4188-a925-77117155617b"
   },
   "source": [
    "## Step-9: Copy output to final output dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
    "outputId": "31f09b58-7b2d-48bb-9dac-bc0ba9625c01"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Copied output from 'output/05_embeddings_out' --> 'output/output_final'\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER_FINAL, ignore_errors=True)\n",
    "shutil.copytree(src=output_folder, dst=MY_CONFIG.OUTPUT_FOLDER_FINAL)\n",
    "\n",
    "print (f\"✅ Copied output from '{output_folder}' --> '{MY_CONFIG.OUTPUT_FOLDER_FINAL}'\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "dpk-2-basic-021-py311",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "06f9b33494984e4885d5aad813d1d2bc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1cb3bbf7d724411cbe9831543a4aecc0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "553f3c16839a49d79591d0fc4862bed6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7053c9606a414e978636a7e241909504": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1cb3bbf7d724411cbe9831543a4aecc0",
      "placeholder": "​",
      "style": "IPY_MODEL_06f9b33494984e4885d5aad813d1d2bc",
      "value": " 10/10 [00:00&lt;00:00, 349.38it/s]"
     }
    },
    "724778729161445c98b187031ae4f67c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "97b603697cfa4b4ea4e6735b6768ca35": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e87e8d3262c54cfaaa8768505edacda3",
       "IPY_MODEL_b78aa40816e44f7fbebcb24ca68818b3",
       "IPY_MODEL_7053c9606a414e978636a7e241909504"
      ],
      "layout": "IPY_MODEL_da0787b239764847a731083997780a85"
     }
    },
    "9d184ed175f0403fb03c2e13dfd04e0a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b78aa40816e44f7fbebcb24ca68818b3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9d184ed175f0403fb03c2e13dfd04e0a",
      "max": 10,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_724778729161445c98b187031ae4f67c",
      "value": 10
     }
    },
    "c0eb5bc8f6ee427ca42204b3c56f9a4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "da0787b239764847a731083997780a85": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e87e8d3262c54cfaaa8768505edacda3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_553f3c16839a49d79591d0fc4862bed6",
      "placeholder": "​",
      "style": "IPY_MODEL_c0eb5bc8f6ee427ca42204b3c56f9a4e",
      "value": "Fetching 10 files: 100%"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}