dpk-intro/dpk_intro_1_python.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866",
      "metadata": {
        "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866"
      },
      "source": [
        "# Data Prep Kit Demo 1 - Python version\n",
        "\n",
        "This notebook will introduce DPK and showcase some of it's capabilities.\n",
        "\n",
        "Here is the workflow\n",
        "\n",
        "![](https://raw.githubusercontent.com/sujee/data-prep-kit-examples/main/media/data-prep-kit-3-workflow.png)\n",
        "\n",
        "References\n",
        "- [Data prep kit](https://ibm.github.io/data-prep-kit)  is an open source framework that helps with data wrangling.\n",
        "- [github repo](https://github.com/IBM/data-prep-kit)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b15976e3",
      "metadata": {
        "id": "b15976e3"
      },
      "source": [
        "## How to run this notebook\n",
        "\n",
        "Two options:\n",
        "\n",
        "- **Option 1 - Google Colab:** easiest option.  no setup required.  Click this link to open this on google colab.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sujee/data-prep-kit-examples/blob/main/dpk-intro/dpk_intro_1_python.ipynb)\n",
        "- **Option 2 - Local python dev environment:**  Setup using this guide: [setting up python dev environment](../setup-python-dev-env.md)\n",
        "\n",
        "The notebook will work as in both environments"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eb8b0d5c",
      "metadata": {
        "id": "eb8b0d5c"
      },
      "source": [
        "## Step-1: Inspect the Data\n",
        "\n",
        "We will use simple PDFs about Solar system.  The files are [here](https://github.com/sujee/data-prep-kit-examples/tree/main/data/solar-system)\n",
        "\n",
        "- [earth.pdf](https://github.com/sujee/data-prep-kit-examples/blob/main/data/solar-system/earth.pdf)\n",
        "- [mars.pdf](https://github.com/sujee/data-prep-kit-examples/blob/main/data/solar-system/mars.pdf)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "39a0ab6e",
      "metadata": {
        "id": "39a0ab6e"
      },
      "source": [
        "## Step-2: Figure out Runtime Environment\n",
        "\n",
        "### 2.1 - Determine runtime\n",
        "\n",
        "Determine if we are running on Google colab or local python environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "1fe354b7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1fe354b7",
        "outputId": "0a38a7b5-238e-433a-c378-78444908aa8a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "NOT in Colab\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "if os.getenv(\"COLAB_RELEASE_TAG\"):\n",
        "   print(\"Running in Colab\")\n",
        "   RUNNING_IN_COLAB = True\n",
        "else:\n",
        "   print(\"NOT in Colab\")\n",
        "   RUNNING_IN_COLAB = False"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8e7c104b",
      "metadata": {
        "id": "8e7c104b"
      },
      "source": [
        "### 2.2 -Download Data if running on Google Colab"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "3309799e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3309799e",
        "outputId": "9b44b764-d284-4da1-ad55-f08d5c9c0f89"
      },
      "outputs": [],
      "source": [
        "if RUNNING_IN_COLAB:\n",
        "    !mkdir -p 'input'\n",
        "    !wget -O 'input/earth.pdf'  'https://raw.githubusercontent.com/sujee/data-prep-kit-examples/main/data/solar-system/earth.pdf'\n",
        "    !wget -O 'input/mars.pdf'  'https://raw.githubusercontent.com/sujee/data-prep-kit-examples/main/data/solar-system/mars.pdf'\n",
        "    !wget -O 'utils.py'  'https://raw.githubusercontent.com/sujee/data-prep-kit-examples/main/utils.py'"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a5dc2b68",
      "metadata": {
        "id": "a5dc2b68"
      },
      "source": [
        "### 2.3 - Install dependencies if running on Google Colab"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "1fcec577",
      "metadata": {
        "id": "1fcec577"
      },
      "outputs": [],
      "source": [
        "if RUNNING_IN_COLAB:\n",
        "    ! pip install  --default-timeout=100  \\\n",
        "        data-prep-toolkit-transforms==0.2.1 \\\n",
        "        data-prep-toolkit-transforms-ray==0.2.1 \\\n",
        "        deepsearch-toolkit"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "243322b8",
      "metadata": {
        "id": "243322b8"
      },
      "source": [
        "### 2.4 - Restart Runtime\n",
        "\n",
        "After installing dependencies, be sure <font color=\"red\">restart runtime</font>, so libraries will be loaded\n",
        "\n",
        "You do this by going to **`Runtime --> Restart Session`**\n",
        "\n",
        "Then you can continue to the next step (no need to re-run the notebook)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e8b10be1",
      "metadata": {
        "id": "e8b10be1"
      },
      "source": [
        "## Step-3: Configuration"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "356c66f7",
      "metadata": {
        "id": "356c66f7"
      },
      "source": [
        "### 3.1 - Basic Config"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "e4YMZrBuFycl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e4YMZrBuFycl",
        "outputId": "42a9edae-205f-4dce-cd4e-a159bd8f620b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "NOT in Colab\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "if os.getenv(\"COLAB_RELEASE_TAG\"):\n",
        "   print(\"Running in Colab\")\n",
        "   RUNNING_IN_COLAB = True\n",
        "else:\n",
        "   print(\"NOT in Colab\")\n",
        "   RUNNING_IN_COLAB = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "33345487",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "33345487",
        "outputId": "79b40d76-b4dd-48ea-9638-461c78a637a1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "MY_CONFIG.RAY_RUNTIME_WORKERS: 2\n",
            "MY_CONFIG.RAY_NUM_CPUS: 0.8\n",
            "MY_CONFIG.RAY_MEMORY_GB: 2\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "## Configuration\n",
        "class MyConfig:\n",
        "    pass\n",
        "\n",
        "MY_CONFIG = MyConfig ()\n",
        "\n",
        "if RUNNING_IN_COLAB:\n",
        "  MY_CONFIG.INPUT_DATA_DIR = 'input'\n",
        "else:\n",
        "  MY_CONFIG.INPUT_DATA_DIR = os.path.join (os.path.abspath (''), '..', 'data', 'solar-system')\n",
        "  \n",
        "MY_CONFIG.OUTPUT_FOLDER = \"output\"\n",
        "MY_CONFIG.OUTPUT_FOLDER_FINAL = os.path.join(MY_CONFIG.OUTPUT_FOLDER , \"output_final\")\n",
        "\n",
        "## Embedding model\n",
        "MY_CONFIG.EMBEDDING_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'\n",
        "\n",
        "## RAY CONFIGURATION\n",
        "### For local runs, we can use more parallelism\n",
        "### For google colab, be conservative\n",
        "\n",
        "if RUNNING_IN_COLAB:\n",
        "  MY_CONFIG.RAY_RUNTIME_WORKERS = 2\n",
        "  MY_CONFIG.RAY_NUM_CPUS =  0.3\n",
        "  MY_CONFIG.RAY_MEMORY_GB = 2  # GB\n",
        "else:  # local run\n",
        "  num_cpus_available =  os.cpu_count()\n",
        "  # print (num_cpus_available)\n",
        "  MY_CONFIG.RAY_NUM_CPUS =  0.8\n",
        "  MY_CONFIG.RAY_MEMORY_GB = 2  # GB\n",
        "  # MY_CONFIG.RAY_RUNTIME_WORKERS = num_cpus_available // 3\n",
        "  MY_CONFIG.RAY_RUNTIME_WORKERS = 2\n",
        "\n",
        "print ('MY_CONFIG.RAY_RUNTIME_WORKERS:', MY_CONFIG.RAY_RUNTIME_WORKERS)\n",
        "print ('MY_CONFIG.RAY_NUM_CPUS:', MY_CONFIG.RAY_NUM_CPUS)\n",
        "print ('MY_CONFIG.RAY_MEMORY_GB:', MY_CONFIG.RAY_MEMORY_GB)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "b15e6827",
      "metadata": {
        "id": "b15e6827"
      },
      "outputs": [],
      "source": [
        "## Add parent dir to path\n",
        "import os,sys\n",
        "\n",
        "this_dir = os.path.abspath('')\n",
        "parent_dir = os.path.dirname(this_dir)\n",
        "sys.path.append (os.path.abspath (parent_dir))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "72510ae6-48b0-4b88-9e13-a623281c3a63",
      "metadata": {
        "id": "72510ae6-48b0-4b88-9e13-a623281c3a63"
      },
      "source": [
        "### 3.2 - Setup input/outpur directories"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "60ac8bee-0960-4309-b225-d7a211b14262",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "60ac8bee-0960-4309-b225-d7a211b14262",
        "outputId": "5c305d54-1c91-455d-d0e2-b514b61a068b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Cleared output directory\n"
          ]
        }
      ],
      "source": [
        "import os, sys\n",
        "import shutil\n",
        "\n",
        "if not os.path.exists(MY_CONFIG.INPUT_DATA_DIR ):\n",
        "    raise Exception (f\"❌ Input folder MY_CONFIG.INPUT_DATA_DIR = '{MY_CONFIG.INPUT_DATA_DIR}' not found\")\n",
        "\n",
        "output_parquet_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '01_parquet_out')\n",
        "output_chunk_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '02_chunk_out')\n",
        "output_docid_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '03_docid_out')\n",
        "output_exact_dedupe_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '04_exact_dedupe_out')\n",
        "output_fuzzy_dedupe_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '05_fuzzy_dedupe_out')\n",
        "output_embeddings_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '06_embeddings_out')\n",
        "\n",
        "## clear output folder\n",
        "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER, ignore_errors=True)\n",
        "shutil.os.makedirs(MY_CONFIG.OUTPUT_FOLDER, exist_ok=True)\n",
        "\n",
        "print (\"✅ Cleared output directory\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb",
      "metadata": {
        "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb"
      },
      "source": [
        "## Step-4: pdf2parquet -  Convert data from PDF to Parquet\n",
        "\n",
        "This step is reading the input folder containing all PDF files and ingest them in a parquet table using the [Docling package](https://github.com/DS4SD/docling).\n",
        "The documents are converted into a JSON format which allows to easily chunk it in the later steps.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c0c574c4-9dc4-4dab-9ad6-b5338207e67a",
      "metadata": {
        "id": "c0c574c4-9dc4-4dab-9ad6-b5338207e67a"
      },
      "source": [
        "### 4.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "482605b2-d814-456d-9195-49a2ec454ef0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "482605b2-d814-456d-9195-49a2ec454ef0",
        "outputId": "90eb1f89-35d1-4b6f-ea34-7667680dd256"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-1: Processing input='/home/sujee/my-stuff/projects/ai-alliance/data-prep-kit-examples/dpk-intro/../data/solar-system' --> output='output/01_parquet_out'\n"
          ]
        }
      ],
      "source": [
        "STAGE = 1\n",
        "\n",
        "input_folder = MY_CONFIG.INPUT_DATA_DIR\n",
        "output_folder =  output_parquet_dir\n",
        "\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b",
      "metadata": {
        "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b"
      },
      "source": [
        "### 4.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "b0cd8ebd-bf71-42d6-a397-8df0c7b66a26",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 625,
          "referenced_widgets": [
            "8226b2522ce446f6bd3a36c4e227370c",
            "7616f1b493e1461c9fd1319fae3bc10b",
            "4f63bfad92b64e7bae18e720376d402d",
            "6957a659451b46dab702c1c62fa9cdd2",
            "2eea7bc810e54eaeb325136352b71e66",
            "ebc626c0750c470db6789b26acf15f60",
            "3077f04af3a9447ab98717bd3131cd8f",
            "709685da1c6c4164bed658357a2191bf",
            "0a1ed94698ca4e4291c553929e0ca66c",
            "5dbc6889a9c243c5a922f8cc5f1a704c",
            "d6e520e4da004c818031ccfcc3588e5d"
          ]
        },
        "id": "b0cd8ebd-bf71-42d6-a397-8df0c7b66a26",
        "outputId": "e2c85b44-f605-4817-c120-2cdce79e3c84"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:33 INFO - pdf2parquet parameters are : {'artifacts_path': None, 'contents_type': <pdf2parquet_contents_types.JSON: 'application/json'>, 'do_table_structure': True, 'do_ocr': True, 'double_precision': 8}\n",
            "00:39:33 INFO - pipeline id pipeline_id\n",
            "00:39:33 INFO - code location None\n",
            "00:39:33 INFO - data factory data_ is using local data access: input_folder - /home/sujee/my-stuff/projects/ai-alliance/data-prep-kit-examples/dpk-intro/../data/solar-system output_folder - output/01_parquet_out\n",
            "00:39:33 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:39:33 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
            "00:39:33 INFO - orchestrator pdf2parquet started at 2024-10-20 00:39:33\n",
            "00:39:33 INFO - Number of files is 2, source profile {'max_file_size': 0.055823326110839844, 'min_file_size': 0.0551910400390625, 'total_file_size': 0.11101436614990234}\n",
            "00:39:33 INFO - Initializing models\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3f1d64aee1e64fa2bbe612ec9d9c7f03",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:38 INFO - Completed 1 files (50.0%) in 0.014 min\n",
            "00:39:38 INFO - Completed 2 files (100.0%) in 0.027 min\n",
            "00:39:38 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "00:39:38 INFO - done flushing in 0.0 sec\n",
            "00:39:38 INFO - Completed execution in 0.083 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:1 completed successfully\n",
            "CPU times: user 11.3 s, sys: 1.64 s, total: 12.9 s\n",
            "Wall time: 7.91 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "import ast\n",
        "import os\n",
        "import sys\n",
        "\n",
        "from pdf2parquet_transform import (\n",
        "    pdf2parquet_contents_type_cli_param,\n",
        "    pdf2parquet_contents_types,\n",
        ")\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from pdf2parquet_transform_python import Pdf2ParquetPythonTransformConfiguration\n",
        "\n",
        "from data_processing.utils import GB, ParamsUtils\n",
        "\n",
        "\n",
        "# create parameters\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "ingest_config = {\n",
        "    pdf2parquet_contents_type_cli_param: pdf2parquet_contents_types.JSON,\n",
        "}\n",
        "\n",
        "params = {\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    \"data_files_to_use\": ast.literal_eval(\"['.pdf']\"),\n",
        "}\n",
        "\n",
        "\n",
        "sys.argv = ParamsUtils.dict_to_req(d=(params | ingest_config))\n",
        "# create launcher\n",
        "launcher = PythonTransformLauncher(Pdf2ParquetPythonTransformConfiguration())\n",
        "# launch\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Job failed\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ca790e0",
      "metadata": {
        "id": "5ca790e0"
      },
      "source": [
        "### 4.3 - Inspect Generated output\n",
        "\n",
        "Here we should see one entry per input file processed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "fe59563d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254
        },
        "id": "fe59563d",
        "outputId": "9eb9a350-c7be-4306-a8ae-4d2414d6c271"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Output dimensions (rows x columns)=  (2, 12)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>contents</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>document_id</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename                                           contents  num_pages  \\\n",
              "0   mars.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...          1   \n",
              "1  earth.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...          1   \n",
              "\n",
              "   num_tables  num_doc_elements                           document_id  ext  \\\n",
              "0           0                11  bcca2c02-29d9-4226-b8b2-92a41704ab6b  pdf   \n",
              "1           0                11  7249c642-01d5-431d-8c42-0f2845ea6060  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf  \n",
              "1  2024-10-20T00:39:38.068313          0.820342       earth.pdf  "
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (\"Output dimensions (rows x columns)= \", output_df.shape)\n",
        "\n",
        "output_df.head(5)\n",
        "\n",
        "## To display certain columns\n",
        "#parquet_df[['column1', 'column2', 'column3']].head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e5058a21",
      "metadata": {
        "id": "e5058a21"
      },
      "source": [
        "\n",
        "### 4.4 - Understand the output\n",
        "\n",
        "Here are some interesting attributes to note:\n",
        "\n",
        "- **filename** : original filename\n",
        "- **contents** : text\n",
        "- **document_id**: unique id (UUID) assignd to this document\n",
        "- **hash** : hash of document\n",
        "- **pdf_convert_time** : time to convert this pdf in seconds\n",
        "\n",
        "Let's inspect the **contents** column.  See how the text is being divided up!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "f870e624",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f870e624",
        "outputId": "f70bfa9f-62f8-417d-d91a-30c1f024ccbd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'_name': '',\n",
            " 'description': {'logs': []},\n",
            " 'equations': [],\n",
            " 'figures': [],\n",
            " 'file-info': {'#-pages': 1,\n",
            "               'document-hash': '1a83f43f3a202e3f203c1263e36961ecc45d401aad488f638fc5559a584333b2',\n",
            "               'filename': 'mars.pdf',\n",
            "               'page-hashes': [{'hash': '551fe7a9bde2a9302f150c0a79a13fcc0868fcf73ac6afb80be645c1174734a0',\n",
            "                                'model': 'default',\n",
            "                                'page': 1}]},\n",
            " 'footnotes': [],\n",
            " 'main-text': [{'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.35137939,\n",
            "                                   654.45184326,\n",
            "                                   169.88169861,\n",
            "                                   667.98492432],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 4]}],\n",
            "                'text': 'Mars',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.09541321,\n",
            "                                   630.68127441,\n",
            "                                   210.66503906,\n",
            "                                   642.34405518],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 12]}],\n",
            "                'text': 'Solar System',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [132.84518433,\n",
            "                                   588.96014404,\n",
            "                                   479.40917969,\n",
            "                                   623.02520752],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 205]}],\n",
            "                'text': 'Our solar system is a vast and fascinating expanse, '\n",
            "                        'comprising eight planets, five dwarf planets, '\n",
            "                        'numerous moons, asteroids, comets, and other '\n",
            "                        'celestial bodies. At its center lies the star we call '\n",
            "                        'the Sun.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [133.18510437,\n",
            "                                   570.83258057,\n",
            "                                   374.99838257,\n",
            "                                   581.07043457],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 54]}],\n",
            "                'text': 'For more details about the Solar system see Chapter '\n",
            "                        '1.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.22866821,\n",
            "                                   542.98168945,\n",
            "                                   163.86282349,\n",
            "                                   554.45288086],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 4]}],\n",
            "                'text': 'Mars',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [132.87440491,\n",
            "                                   500.84011841,\n",
            "                                   477.48345947,\n",
            "                                   534.55810547],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 196]}],\n",
            "                'text': 'Mars, the fourth planet from the Sun, is a cold, '\n",
            "                        'desert world with a thin atmosphere composed '\n",
            "                        'primarily of carbon dioxide. Its reddish hue comes '\n",
            "                        'from iron oxide, or rust, prevalent on its surface.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.2026062,\n",
            "                                   482.90710449,\n",
            "                                   237.04431152,\n",
            "                                   493.07443237],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 23]}],\n",
            "                'text': 'Basic facts about Mars:',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   453.019104,\n",
            "                                   477.48171997,\n",
            "                                   474.9703064],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 78]}],\n",
            "                'text': '· Distance from the Sun: Average of 228 million '\n",
            "                        'kilometers (142 million miles)',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   440.79351807,\n",
            "                                   431.73287964,\n",
            "                                   451.2142334],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 64]}],\n",
            "                'text': '· Rotation Period: 24.6 hours (one Martian day - '\n",
            "                        'called a \"sol\")',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   429.10913086,\n",
            "                                   365.9559021,\n",
            "                                   438.83737183],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 44]}],\n",
            "                'text': '· Moons: Two small moons, Phobos and Deimos.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Page-footer',\n",
            "                'prov': [{'bbox': [303.13299561,\n",
            "                                   87.20314026,\n",
            "                                   308.11428833,\n",
            "                                   96.51646423],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 1]}],\n",
            "                'text': '1',\n",
            "                'type': 'page-footer'}],\n",
            " 'page-dimensions': [{'height': 792.0, 'page': 1, 'width': 612.0}],\n",
            " 'page-footers': [],\n",
            " 'page-headers': [],\n",
            " 'tables': [],\n",
            " 'type': 'pdf-document'}\n"
          ]
        }
      ],
      "source": [
        "import pprint\n",
        "import json\n",
        "\n",
        "pprint.pprint (json.loads(output_df.iloc[0, ]['contents']))\n",
        "# json.loads(output_df.iloc[0, ]['contents'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "e1a10c2d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e1a10c2d",
        "outputId": "300e7688-692a-4039-c4a4-a86887d9138b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'_name': '',\n",
            " 'description': {'logs': []},\n",
            " 'equations': [],\n",
            " 'figures': [],\n",
            " 'file-info': {'#-pages': 1,\n",
            "               'document-hash': '7401ae81637dbb89e7040dcd5945bbfb75ff8648bb761c69f8a1595e86538748',\n",
            "               'filename': 'earth.pdf',\n",
            "               'page-hashes': [{'hash': 'ca802e4bd5a3301792808caea2a47db51f0520888875b77fc230c99ee851c19b',\n",
            "                                'model': 'default',\n",
            "                                'page': 1}]},\n",
            " 'footnotes': [],\n",
            " 'main-text': [{'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.30961609,\n",
            "                                   654.45184326,\n",
            "                                   174.04208374,\n",
            "                                   667.93347168],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 5]}],\n",
            "                'text': 'Earth',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.12528992,\n",
            "                                   630.69073486,\n",
            "                                   210.66503906,\n",
            "                                   642.27935791],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 12]}],\n",
            "                'text': 'Solar System',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [132.87112427,\n",
            "                                   588.96014404,\n",
            "                                   479.40917969,\n",
            "                                   623.04595947],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 205]}],\n",
            "                'text': 'Our solar system is a vast and fascinating expanse, '\n",
            "                        'comprising eight planets, five dwarf planets, '\n",
            "                        'numerous moons, asteroids, comets, and other '\n",
            "                        'celestial bodies. At its center lies the star we call '\n",
            "                        'the Sun.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [133.20942688,\n",
            "                                   570.81555176,\n",
            "                                   375.57919312,\n",
            "                                   581.08459473],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 54]}],\n",
            "                'text': 'For more details about our Solar system see Chapter '\n",
            "                        '1.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Section-header',\n",
            "                'prov': [{'bbox': [133.15542603,\n",
            "                                   542.98168945,\n",
            "                                   167.32983398,\n",
            "                                   554.36669922],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 5]}],\n",
            "                'text': 'Earth',\n",
            "                'type': 'subtitle-level-1'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [132.91053772,\n",
            "                                   512.46295166,\n",
            "                                   477.84887695,\n",
            "                                   534.48431396],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 107]}],\n",
            "                'text': \"Earth is the third planet from the Sun. It's our home \"\n",
            "                        'planet. Earth is the only place we know of with life.',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Text',\n",
            "                'prov': [{'bbox': [133.30151367,\n",
            "                                   494.86206055,\n",
            "                                   240.17156982,\n",
            "                                   505.07229614],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 24]}],\n",
            "                'text': 'Basic facts about Earth:',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   464.97409058,\n",
            "                                   477.47979736,\n",
            "                                   487.02810669],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 79]}],\n",
            "                'text': '· Distance from the Sun: Average of 149.6 million '\n",
            "                        'kilometers (93 million miles)',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   452.86901855,\n",
            "                                   317.90722656,\n",
            "                                   463.24041748],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 37]}],\n",
            "                'text': '· Rotation Period: 24 hours (one day)',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'List-item',\n",
            "                'prov': [{'bbox': [145.94500732,\n",
            "                                   440.71496582,\n",
            "                                   396.66357422,\n",
            "                                   451.19915771],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 52]}],\n",
            "                'text': '· Moons: One moon, called Luna or simply \"the Moon\".',\n",
            "                'type': 'paragraph'},\n",
            "               {'name': 'Page-footer',\n",
            "                'prov': [{'bbox': [303.13299561,\n",
            "                                   87.20314026,\n",
            "                                   308.11428833,\n",
            "                                   96.53633118],\n",
            "                          'page': 1,\n",
            "                          'span': [0, 1]}],\n",
            "                'text': '1',\n",
            "                'type': 'page-footer'}],\n",
            " 'page-dimensions': [{'height': 792.0, 'page': 1, 'width': 612.0}],\n",
            " 'page-footers': [],\n",
            " 'page-headers': [],\n",
            " 'tables': [],\n",
            " 'type': 'pdf-document'}\n"
          ]
        }
      ],
      "source": [
        "pprint.pprint (json.loads(output_df.iloc[1, ]['contents']))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "72274586",
      "metadata": {
        "id": "72274586"
      },
      "source": [
        "##  Step-5: Doc chunks\n",
        "\n",
        "In the previous step, we have extracted text from oru PDFs.  But we have the content of entire file as 'one row' in our parquet output.\n",
        "\n",
        "In this step, we are going to split the documents in chunks, according to their layout segmentation.\n",
        "\n",
        "This transform uses [Quackling](https://github.com/DS4SD/quackling) `HierarchicalChunker`\n",
        "to chunk according to the document layout segmentation, i.e. respecting the original document components as paragraphs, tables, enumerations, etc.\n",
        "It relies on documents converted with the Docling library in the [pdf2parquet transform](https://github.com/IBM/data-prep-kit/blob/dev/transforms/language/pdf2parquet/python/README.md) using the option `contents_type: \"application/json\"`,\n",
        "which provides the required JSON structure."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "96198fa6",
      "metadata": {
        "id": "96198fa6"
      },
      "source": [
        "### 5.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "305f00a3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "305f00a3",
        "outputId": "a787385b-214a-41b2-975d-0d3c5529c2c4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-2: Processing input='output/01_parquet_out' --> output='output/02_chunk_out'\n"
          ]
        }
      ],
      "source": [
        "STAGE = 2\n",
        "\n",
        "input_folder = output_parquet_dir # previous output folder is the input folder for the current stage\n",
        "output_folder =  output_chunk_dir\n",
        "\n",
        "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
        "\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "369f2cd1",
      "metadata": {
        "id": "369f2cd1"
      },
      "source": [
        "### 5.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "5b7b18d5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5b7b18d5",
        "outputId": "cb338503-3dca-45bd-a60a-bd214843a97b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:39 INFO - doc_chunk parameters are : {'chunking_type': <chunking_types.DL_JSON: 'dl_json'>, 'content_column_name': 'contents', 'doc_id_column_name': 'document_id', 'dl_min_chunk_len': None, 'output_chunk_column_name': 'contents', 'output_source_doc_id_column_name': 'source_document_id', 'output_jsonpath_column_name': 'doc_jsonpath', 'output_pageno_column_name': 'page_number', 'output_bbox_column_name': 'bbox'}\n",
            "00:39:39 INFO - pipeline id pipeline_id\n",
            "00:39:39 INFO - code location None\n",
            "00:39:39 INFO - data factory data_ is using local data access: input_folder - output/01_parquet_out output_folder - output/02_chunk_out\n",
            "00:39:39 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:39:39 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
            "00:39:39 INFO - orchestrator doc_chunk started at 2024-10-20 00:39:39\n",
            "00:39:39 INFO - Number of files is 2, source profile {'max_file_size': 0.02239513397216797, 'min_file_size': 0.02167987823486328, 'total_file_size': 0.04407501220703125}\n",
            "00:39:39 INFO - Completed 1 files (50.0%) in 0.0 min\n",
            "00:39:39 INFO - Completed 2 files (100.0%) in 0.0 min\n",
            "00:39:39 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "00:39:39 INFO - done flushing in 0.0 sec\n",
            "00:39:39 INFO - Completed execution in 0.0 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:2 completed successfully\n",
            "CPU times: user 868 ms, sys: 89.5 ms, total: 957 ms\n",
            "Wall time: 952 ms\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from doc_chunk_transform_python import DocChunkPythonTransformConfiguration\n",
        "\n",
        "\n",
        "# Prepare the commandline params\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "params = {\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    # doc_chunk arguments\n",
        "    # ...\n",
        "}\n",
        "\n",
        "# Pass the commandline params\n",
        "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
        "\n",
        "# create launcher\n",
        "launcher = PythonTransformLauncher(DocChunkPythonTransformConfiguration())\n",
        "# launch\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Job failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "213afdf6",
      "metadata": {
        "id": "213afdf6"
      },
      "source": [
        "### 5.3 - Inspect Generated output\n",
        "\n",
        "We would see documents are split into many chunks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "d8138d43",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 893
        },
        "id": "d8138d43",
        "outputId": "0d08e0a6-e743-44d9-b8f1-eec98b222a92"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Files processed : 2\n",
            "Chunks created : 8\n",
            "Input data dimensions (rows x columns)=  (2, 12)\n",
            "Output data dimensions (rows x columns)=  (8, 16)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "      <th>source_document_id</th>\n",
              "      <th>contents</th>\n",
              "      <th>doc_jsonpath</th>\n",
              "      <th>page_number</th>\n",
              "      <th>bbox</th>\n",
              "      <th>document_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.84518433, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Solar System\\nFor more details about the Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
              "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
              "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
              "0   mars.pdf          1           0                11  pdf   \n",
              "1   mars.pdf          1           0                11  pdf   \n",
              "2   mars.pdf          1           0                11  pdf   \n",
              "3   mars.pdf          1           0                11  pdf   \n",
              "4  earth.pdf          1           0                11  pdf   \n",
              "5  earth.pdf          1           0                11  pdf   \n",
              "6  earth.pdf          1           0                11  pdf   \n",
              "7  earth.pdf          1           0                11  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "3  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "7  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \\\n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "1  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "2  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "3  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "4  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "5  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "6  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "7  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "\n",
              "                     source_document_id  \\\n",
              "0  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "1  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "2  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "3  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "4  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "5  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "6  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "7  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "\n",
              "                                            contents    doc_jsonpath  \\\n",
              "0  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "1  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
              "2  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
              "3  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
              "4  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "5  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
              "6  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
              "7  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
              "\n",
              "   page_number                                               bbox  \\\n",
              "0            1  [132.84518433, 588.96014404, 479.40917969, 623...   \n",
              "1            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
              "2            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
              "3            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
              "4            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
              "5            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
              "6            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
              "7            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
              "\n",
              "                                         document_id  \n",
              "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...  \n",
              "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...  \n",
              "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...  \n",
              "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...  \n",
              "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...  \n",
              "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...  \n",
              "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...  \n",
              "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...  "
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (f\"Files processed : {input_df.shape[0]:,}\")\n",
        "print (f\"Chunks created : {output_df.shape[0]:,}\")\n",
        "\n",
        "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
        "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
        "\n",
        "output_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9e9ca75c",
      "metadata": {
        "id": "9e9ca75c"
      },
      "source": [
        "### 5.4 - Understanding the Output\n",
        "\n",
        "Here we see 2 PDF files are split into 6 chunks.  Basically we see the documents are being split along 'natural boundaris' - paragraphs and bullet points\n",
        "\n",
        "See how **document_id** is carried throughout.  This helps us identify original documents.\n",
        "\n",
        "Also note **contents** is now plain text (not JSON as before)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "3090c950",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "3090c950",
        "outputId": "cf9bd956-7b31-42bc-ef77-9ebded8ba08e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>contents</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Solar System\\nFor more details about the Solar...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename                                           contents\n",
              "0   mars.pdf  Solar System\\nOur solar system is a vast and f...\n",
              "1   mars.pdf  Solar System\\nFor more details about the Solar...\n",
              "2   mars.pdf  Mars\\nMars, the fourth planet from the Sun, is...\n",
              "3   mars.pdf  Basic facts about Mars:\\n· Distance from the S...\n",
              "4  earth.pdf  Solar System\\nOur solar system is a vast and f...\n",
              "5  earth.pdf  Solar System\\nFor more details about our Solar...\n",
              "6  earth.pdf  Earth\\nEarth is the third planet from the Sun....\n",
              "7  earth.pdf  Earth\\nBasic facts about Earth:\\n· Distance fr..."
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output_df[['filename', 'contents']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "d5f151ae",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d5f151ae",
        "outputId": "2b48675c-328d-4d24-d689-ad77231ef4b7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "========== mars.pdf ===========\n",
            "-------Chunk 0------\n",
            "Solar System\n",
            "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Solar System\n",
            "For more details about the Solar system see Chapter 1.\n",
            "-------\n",
            "-------Chunk 2------\n",
            "Mars\n",
            "Mars, the fourth planet from the Sun, is a cold, desert world with a thin atmosphere composed primarily of carbon dioxide. Its reddish hue comes from iron oxide, or rust, prevalent on its surface.\n",
            "-------\n",
            "-------Chunk 3------\n",
            "Basic facts about Mars:\n",
            "· Distance from the Sun: Average of 228 million kilometers (142 million miles)\n",
            "· Rotation Period: 24.6 hours (one Martian day - called a \"sol\")\n",
            "· Moons: Two small moons, Phobos and Deimos.\n",
            "-------\n",
            "========== earth.pdf ===========\n",
            "-------Chunk 0------\n",
            "Solar System\n",
            "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Solar System\n",
            "For more details about our Solar system see Chapter 1.\n",
            "-------\n",
            "-------Chunk 2------\n",
            "Earth\n",
            "Earth is the third planet from the Sun. It's our home planet. Earth is the only place we know of with life.\n",
            "-------\n",
            "-------Chunk 3------\n",
            "Earth\n",
            "Basic facts about Earth:\n",
            "· Distance from the Sun: Average of 149.6 million kilometers (93 million miles)\n",
            "· Rotation Period: 24 hours (one day)\n",
            "· Moons: One moon, called Luna or simply \"the Moon\".\n",
            "-------\n"
          ]
        }
      ],
      "source": [
        "for f in output_df['filename'].unique():\n",
        "    print ('==========' , f, '===========')\n",
        "    chunks = output_df[output_df['filename'] == f]['contents']\n",
        "    for idx , chunk in enumerate(chunks):\n",
        "        print (f'-------Chunk {idx}------\\n{chunk}\\n-------')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7ad1c60d",
      "metadata": {},
      "source": [
        "## Step-6:  DOC ID generation of Chunks\n",
        "\n",
        "This transform annotates documents with document \"ids\". It supports the following transformations of the original data:\n",
        "\n",
        " - Adding document hash: this enables the addition of a document hash-based id to the data. The hash is calculated with `hashlib.sha256(doc.encode(\"utf-8\")).hexdigest()`. To enable this annotation, set **hash_column** to the name of the column, where you want to store it.\n",
        " - Adding integer document id: this allows the addition of an integer document id to the data that is unique across all rows in all tables provided to the transform() method. To enable this annotation, set **int_id_column** to the name of the column, where you want to store it.\n",
        "\n",
        "**This is a pre-requisite for fuzzy dedup** in the pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1afaa0fd",
      "metadata": {},
      "source": [
        "### 6.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "6ffd6f54",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-3: Processing input='output/02_chunk_out' --> output='output/03_docid_out'\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Input for this stage is the output of exact dedeup component\n",
        "# output of this component makes it possible for fdedup component to run on data.\n",
        "\n",
        "STAGE  = 3\n",
        "\n",
        "input_folder = output_chunk_dir\n",
        "output_folder =  output_docid_dir\n",
        "\n",
        "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
        "\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f78a51b7",
      "metadata": {},
      "source": [
        "### 6.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "5fc77557",
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:40 INFO - Doc id parameters are : {'doc_column': 'contents', 'hash_column': 'chunk_hash', 'int_column': 'chunk_id', 'start_id': 0}\n",
            "00:39:40 INFO - pipeline id pipeline_id\n",
            "00:39:40 INFO - code location None\n",
            "00:39:40 INFO - data factory data_ is using local data access: input_folder - output/02_chunk_out output_folder - output/03_docid_out\n",
            "00:39:40 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:39:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
            "00:39:40 INFO - orchestrator doc_id started at 2024-10-20 00:39:40\n",
            "00:39:40 INFO - Number of files is 2, source profile {'max_file_size': 0.008975982666015625, 'min_file_size': 0.008897781372070312, 'total_file_size': 0.017873764038085938}\n",
            "00:39:40 INFO - Completed 1 files (50.0%) in 0.0 min\n",
            "00:39:40 INFO - Completed 2 files (100.0%) in 0.0 min\n",
            "00:39:40 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "00:39:40 INFO - done flushing in 0.0 sec\n",
            "00:39:40 INFO - Completed execution in 0.0 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:3 completed successfully\n",
            "CPU times: user 13.3 ms, sys: 2.33 ms, total: 15.6 ms\n",
            "Wall time: 12.5 ms\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from doc_id_transform_python import DocIDPythonTransformRuntimeConfiguration\n",
        "\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "params = {\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    # orchestrator\n",
        "    # doc id configuration\n",
        "    \"doc_id_doc_column\": \"contents\",\n",
        "    \"doc_id_hash_column\": \"chunk_hash\",\n",
        "    \"doc_id_int_column\": \"chunk_id\",\n",
        "}\n",
        "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
        "\n",
        "# launch\n",
        "\n",
        "launcher = PythonTransformLauncher(DocIDPythonTransformRuntimeConfiguration())\n",
        "\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Job failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a9a8c1fa",
      "metadata": {},
      "source": [
        "### 6.3 - Inspect Generated output\n",
        "\n",
        "You will notice we have two extra columns\n",
        "\n",
        "- **hash_column**\n",
        "- **int_id_column**\n",
        "\n",
        "But still the same number or rows as before"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "da9adede",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input data dimensions (rows x columns)=  (8, 16)\n",
            "Output data dimensions (rows x columns)=  (8, 18)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "      <th>source_document_id</th>\n",
              "      <th>contents</th>\n",
              "      <th>doc_jsonpath</th>\n",
              "      <th>page_number</th>\n",
              "      <th>bbox</th>\n",
              "      <th>document_id</th>\n",
              "      <th>chunk_hash</th>\n",
              "      <th>chunk_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.84518433, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Solar System\\nFor more details about the Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
              "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
              "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
              "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
              "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
              "0   mars.pdf          1           0                11  pdf   \n",
              "1   mars.pdf          1           0                11  pdf   \n",
              "2   mars.pdf          1           0                11  pdf   \n",
              "3   mars.pdf          1           0                11  pdf   \n",
              "4  earth.pdf          1           0                11  pdf   \n",
              "5  earth.pdf          1           0                11  pdf   \n",
              "6  earth.pdf          1           0                11  pdf   \n",
              "7  earth.pdf          1           0                11  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "3  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "7  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \\\n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "1  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "2  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "3  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "4  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "5  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "6  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "7  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "\n",
              "                     source_document_id  \\\n",
              "0  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "1  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "2  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "3  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "4  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "5  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "6  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "7  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "\n",
              "                                            contents    doc_jsonpath  \\\n",
              "0  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "1  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
              "2  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
              "3  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
              "4  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "5  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
              "6  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
              "7  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
              "\n",
              "   page_number                                               bbox  \\\n",
              "0            1  [132.84518433, 588.96014404, 479.40917969, 623...   \n",
              "1            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
              "2            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
              "3            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
              "4            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
              "5            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
              "6            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
              "7            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
              "\n",
              "                                         document_id  \\\n",
              "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
              "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...   \n",
              "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...   \n",
              "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...   \n",
              "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
              "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...   \n",
              "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...   \n",
              "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...   \n",
              "\n",
              "                                          chunk_hash  chunk_id  \n",
              "0  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         4  \n",
              "1  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...         5  \n",
              "2  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6  \n",
              "3  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7  \n",
              "4  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0  \n",
              "5  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1  \n",
              "6  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2  \n",
              "7  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...         3  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
        "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
        "\n",
        "output_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4692975c-49ff-41ae-810e-0f5bc0bbdc53",
      "metadata": {
        "id": "4692975c-49ff-41ae-810e-0f5bc0bbdc53"
      },
      "source": [
        "## Step-7: Exact Dedup\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5acfd3a2-a236-4143-bcfc-15804f1da7fe",
      "metadata": {
        "id": "5acfd3a2-a236-4143-bcfc-15804f1da7fe"
      },
      "source": [
        "### 7.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "4c7a1b94",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4c7a1b94",
        "outputId": "2a135853-c54f-4aa4-ffc4-83c2bc7a68ce"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-4: Processing input='output/03_docid_out' --> output='output/04_exact_dedupe_out'\n"
          ]
        }
      ],
      "source": [
        "STAGE  = 4\n",
        "\n",
        "input_folder = output_docid_dir # previous output folder is the input folder for the current stage\n",
        "output_folder =  output_exact_dedupe_dir\n",
        "\n",
        "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
        "\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3661cb37-39c7-4b09-a784-925bfa9eaf1e",
      "metadata": {
        "id": "3661cb37-39c7-4b09-a784-925bfa9eaf1e"
      },
      "source": [
        "### 7.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "a624b2b2-faad-4325-ac7d-53a840f564ef",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a624b2b2-faad-4325-ac7d-53a840f564ef",
        "outputId": "b9b3de92-4304-4540-dfba-a4549fa157eb"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:40 INFO - exact dedup params are {'doc_column': 'contents', 'doc_id_column': 'chunk_hash', 'use_snapshot': False, 'snapshot_directory': None}\n",
            "00:39:40 INFO - pipeline id pipeline_id\n",
            "00:39:40 INFO - code location None\n",
            "00:39:40 INFO - data factory data_ is using local data access: input_folder - output/03_docid_out output_folder - output/04_exact_dedupe_out\n",
            "00:39:40 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:39:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
            "00:39:40 INFO - orchestrator ededup started at 2024-10-20 00:39:40\n",
            "00:39:40 INFO - Number of files is 2, source profile {'max_file_size': 0.010180473327636719, 'min_file_size': 0.010101318359375, 'total_file_size': 0.02028179168701172}\n",
            "00:39:40 INFO - Starting from the beginning\n",
            "00:39:40 INFO - Completed 1 files (50.0%) in 0.0 min\n",
            "00:39:40 INFO - Completed 2 files (100.0%) in 0.0 min\n",
            "00:39:40 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "00:39:40 INFO - done flushing in 0.0 sec\n",
            "00:39:40 INFO - Completed execution in 0.0 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:4 completed successfully\n",
            "CPU times: user 16.9 ms, sys: 1.41 ms, total: 18.3 ms\n",
            "Wall time: 15.2 ms\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from ededup_transform_python import EdedupPythonTransformRuntimeConfiguration\n",
        "\n",
        "\n",
        "# Prepare the commandline params\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "params = {\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    # ededup parameters\n",
        "    \"ededup_doc_column\": \"contents\",\n",
        "    \"ededup_doc_id_column\": \"chunk_hash\",\n",
        "}\n",
        "\n",
        "# Pass the commandline params\n",
        "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
        "\n",
        "# create launcher\n",
        "launcher = PythonTransformLauncher(EdedupPythonTransformRuntimeConfiguration())\n",
        "# launch\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Job failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eaf1c3c3",
      "metadata": {
        "id": "eaf1c3c3"
      },
      "source": [
        "### 7.3 - Inspect Generated output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "d824ebf6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 358
        },
        "id": "d824ebf6",
        "outputId": "14aa660f-6f1a-4f93-9b61-5f8f8adcf3fe"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input data dimensions (rows x columns)=  (8, 18)\n",
            "Output data dimensions (rows x columns)=  (7, 19)\n",
            "Input chunks before exact dedupe : 8\n",
            "Output chunks after exact dedupe : 7\n",
            "Duplicate chunks removed :   1\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "      <th>source_document_id</th>\n",
              "      <th>contents</th>\n",
              "      <th>doc_jsonpath</th>\n",
              "      <th>page_number</th>\n",
              "      <th>bbox</th>\n",
              "      <th>document_id</th>\n",
              "      <th>chunk_hash</th>\n",
              "      <th>chunk_id</th>\n",
              "      <th>removed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Solar System\\nFor more details about the Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.18510437, 570.83258057, 374.99838257, 581...</td>\n",
              "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
              "      <td>dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...</td>\n",
              "      <td>5</td>\n",
              "      <td>[44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>6</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>0</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "      <td>1</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "      <td>2</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.30151367, 494.86206055, 240.17156982, 505...</td>\n",
              "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
              "      <td>189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...</td>\n",
              "      <td>3</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
              "0   mars.pdf          1           0                11  pdf   \n",
              "1   mars.pdf          1           0                11  pdf   \n",
              "2   mars.pdf          1           0                11  pdf   \n",
              "3  earth.pdf          1           0                11  pdf   \n",
              "4  earth.pdf          1           0                11  pdf   \n",
              "5  earth.pdf          1           0                11  pdf   \n",
              "6  earth.pdf          1           0                11  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "2  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "3  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "5  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "6  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \\\n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "1  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "2  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "3  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "4  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "5  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "6  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "\n",
              "                     source_document_id  \\\n",
              "0  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "1  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "2  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "3  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "4  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "5  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "6  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "\n",
              "                                            contents    doc_jsonpath  \\\n",
              "0  Solar System\\nFor more details about the Solar...  $.main-text[3]   \n",
              "1  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
              "2  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
              "3  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "4  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
              "5  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
              "6  Earth\\nBasic facts about Earth:\\n· Distance fr...  $.main-text[6]   \n",
              "\n",
              "   page_number                                               bbox  \\\n",
              "0            1  [133.18510437, 570.83258057, 374.99838257, 581...   \n",
              "1            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
              "2            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
              "3            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
              "4            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
              "5            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
              "6            1  [133.30151367, 494.86206055, 240.17156982, 505...   \n",
              "\n",
              "                                         document_id  \\\n",
              "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...   \n",
              "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...   \n",
              "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...   \n",
              "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...   \n",
              "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...   \n",
              "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...   \n",
              "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...   \n",
              "\n",
              "                                          chunk_hash  chunk_id  \\\n",
              "0  dee4c03474c98efdabbadbcc4ce91138c7820f4ac8ff07...         5   \n",
              "1  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6   \n",
              "2  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7   \n",
              "3  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0   \n",
              "4  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1   \n",
              "5  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2   \n",
              "6  189a221704d17feeb96b1b1ef60a2a2445459848cd8e8f...         3   \n",
              "\n",
              "                                             removed  \n",
              "0  [44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf567...  \n",
              "1                                                 []  \n",
              "2                                                 []  \n",
              "3                                                 []  \n",
              "4                                                 []  \n",
              "5                                                 []  \n",
              "6                                                 []  "
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
        "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
        "print (f\"Input chunks before exact dedupe : {input_df.shape[0]:,}\")\n",
        "print (f\"Output chunks after exact dedupe : {output_df.shape[0]:,}\")\n",
        "print (\"Duplicate chunks removed :  \", (input_df.shape[0] - output_df.shape[0]))\n",
        "\n",
        "output_df.head(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "82cc9bb0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "82cc9bb0",
        "outputId": "2aff0a5f-8cc7-408c-e1cf-62c0b14b18fb"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>contents</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Solar System\\nFor more details about the Solar...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename                                           contents\n",
              "0   mars.pdf  Solar System\\nFor more details about the Solar...\n",
              "1   mars.pdf  Mars\\nMars, the fourth planet from the Sun, is...\n",
              "2   mars.pdf  Basic facts about Mars:\\n· Distance from the S...\n",
              "3  earth.pdf  Solar System\\nOur solar system is a vast and f...\n",
              "4  earth.pdf  Solar System\\nFor more details about our Solar...\n",
              "5  earth.pdf  Earth\\nEarth is the third planet from the Sun....\n",
              "6  earth.pdf  Earth\\nBasic facts about Earth:\\n· Distance fr..."
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output_df[['filename', 'contents']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "cc61dffa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cc61dffa",
        "outputId": "337b015f-3795-4c45-98a3-03ae817d4dca"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "========== mars.pdf ===========\n",
            "-------Chunk 0------\n",
            "Solar System\n",
            "For more details about the Solar system see Chapter 1.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Mars\n",
            "Mars, the fourth planet from the Sun, is a cold, desert world with a thin atmosphere composed primarily of carbon dioxide. Its reddish hue comes from iron oxide, or rust, prevalent on its surface.\n",
            "-------\n",
            "-------Chunk 2------\n",
            "Basic facts about Mars:\n",
            "· Distance from the Sun: Average of 228 million kilometers (142 million miles)\n",
            "· Rotation Period: 24.6 hours (one Martian day - called a \"sol\")\n",
            "· Moons: Two small moons, Phobos and Deimos.\n",
            "-------\n",
            "========== earth.pdf ===========\n",
            "-------Chunk 0------\n",
            "Solar System\n",
            "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Solar System\n",
            "For more details about our Solar system see Chapter 1.\n",
            "-------\n",
            "-------Chunk 2------\n",
            "Earth\n",
            "Earth is the third planet from the Sun. It's our home planet. Earth is the only place we know of with life.\n",
            "-------\n",
            "-------Chunk 3------\n",
            "Earth\n",
            "Basic facts about Earth:\n",
            "· Distance from the Sun: Average of 149.6 million kilometers (93 million miles)\n",
            "· Rotation Period: 24 hours (one day)\n",
            "· Moons: One moon, called Luna or simply \"the Moon\".\n",
            "-------\n"
          ]
        }
      ],
      "source": [
        "for f in output_df['filename'].unique():\n",
        "    print ('==========' , f, '===========')\n",
        "    chunks = output_df[output_df['filename'] == f]['contents']\n",
        "    for idx , chunk in enumerate(chunks):\n",
        "        print (f'-------Chunk {idx}------\\n{chunk}\\n-------')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "383f40ba",
      "metadata": {
        "id": "383f40ba"
      },
      "source": [
        "### 7.4 - Understanding the output\n",
        "\n",
        "Remember we had 8 chunks initially.  Now we have 7!  One duplicate chunk is removed.\n",
        "\n",
        "If you look at the PDF, the following common paragraph in `earth.pdf` and `mars.pdf`  is removed from one of the documents!  Pretty neat, eh!\n",
        "\n",
        "```text\n",
        "## Solar System\n",
        "\n",
        "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "85309751-8556-41c6-ac32-84acc941bc8d",
      "metadata": {
        "id": "85309751-8556-41c6-ac32-84acc941bc8d"
      },
      "source": [
        "  ## Step-8: Fuzzy Dedup\n",
        "\n",
        "Post exact deduplication, fuzzy deduplication is applied with the goal of removing **very similar** chunks\n",
        "\n",
        "And fuzzy dedupe is only available in RAY version."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fcf574a3-b287-419c-9c86-07b828b41ca6",
      "metadata": {
        "id": "fcf574a3-b287-419c-9c86-07b828b41ca6"
      },
      "source": [
        "### 8.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "id": "9e431c8c-c7c7-48de-ba5f-2c4649c35399",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9e431c8c-c7c7-48de-ba5f-2c4649c35399",
        "outputId": "4450ed63-3b09-42e4-8085-2951e700cf8f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-5: Processing input='output/04_exact_dedupe_out' --> output='output/05_fuzzy_dedupe_out'\n"
          ]
        }
      ],
      "source": [
        "## Input to this component is the output of doc_id generator component.\n",
        "\n",
        "STAGE  = 5\n",
        "\n",
        "input_folder = output_exact_dedupe_dir # previous output folder is the input folder for the current stage\n",
        "output_folder =  output_fuzzy_dedupe_dir\n",
        "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f4c82a8f-b513-4fe5-b172-d41b104b54f3",
      "metadata": {
        "id": "f4c82a8f-b513-4fe5-b172-d41b104b54f3"
      },
      "source": [
        "### 8.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "id": "3864ff77-e9a8-48f7-973b-c3b3aef1a94f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3864ff77-e9a8-48f7-973b-c3b3aef1a94f",
        "outputId": "2baa790d-6944-4d20-f0c1-fc2979eb1686"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:39:40 INFO - fuzzy dedup params are {'doc_column': 'contents', 'id_column': 'chunk_id', 'cluster_column': 'chunk_hash', 'bucket_cpu': 0.3, 'mhash_cpu': 0.3, 'doc_cpu': 0.3, 'num_doc_actors': 1, 'num_minhash_actors': 1, 'num_bucket_actors': 1, 'num_preprocessors': 1, 'num_permutations': 64, 'threshold': 0.7, 'shingles_size': 5, 'delimiters': ' ', 'snapshot_delay': 1, 'use_bucket_snapshot': False, 'use_doc_snapshot': False, 'random_delay_limit': 10, 'worker_options': {'num_cpus': 0.8}}\n",
            "00:39:40 INFO - pipeline id pipeline_id\n",
            "00:39:40 INFO - code location None\n",
            "00:39:40 INFO - number of workers 2 worker options {'num_cpus': 0.8, 'max_restarts': -1}\n",
            "00:39:40 INFO - actor creation delay 0\n",
            "00:39:40 INFO - job details {'job category': 'preprocessing', 'job name': 'fdedup', 'job type': 'ray', 'job id': 'job_id'}\n",
            "00:39:40 INFO - data factory data_ is using local data access: input_folder - output/04_exact_dedupe_out output_folder - output/05_fuzzy_dedupe_out\n",
            "00:39:40 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:39:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
            "00:39:40 INFO - Running locally\n",
            "2024-10-20 00:39:42,380\tINFO worker.py:1744 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - orchestrator started at 2024-10-20 00:39:43\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - Number of files is 2, source profile {'max_file_size': 0.010450363159179688, 'min_file_size': 0.010318756103515625, 'total_file_size': 0.020769119262695312}\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - Cluster resources: {'cpus': 16, 'gpus': 1, 'memory': 10.211402893066406, 'object_store': 5.105701446533203}\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - Number of workers - 2 with {'num_cpus': 0.8, 'max_restarts': -1} each\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - starting run from the beginning\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - continuing from the very beginning\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - Fuzzy: num buckets 8, bucket length 8\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - created 1 bucket actors\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - created 1 minhash actors\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - Table preprocessing uses 1 readers\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:43 INFO - created 1 table processor actors\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:49 INFO - Completed 1 files in 0.097 min\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:49 INFO - Completed 1 files (50.0%)  in 0.097 min. Waiting for completion\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:55 INFO - Completed processing 2 files in 0.197 min\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:55 INFO - creating minhash snapshots\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:56 INFO - minhash snapshots created\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:56 INFO - creating bucket snapshots\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:57 INFO - bucket snapshots created\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:57 INFO - created 1 document actors\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:57 INFO - created 1 bucket processor actors\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:57 INFO - created bucket processor invoker\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:57 INFO - added invoker to bucket collectors\n",
            "\u001b[36m(BucketsHash pid=509820)\u001b[0m 00:39:57 INFO - processing buckets 0 long, 53 short\n",
            "\u001b[36m(BucketsHash pid=509820)\u001b[0m 00:39:57 INFO - Done submitting long buckets\n",
            "\u001b[36m(BucketsHashProcessorInvoker pid=510263)\u001b[0m 00:39:57 INFO - Waiting bucket processing completion. Submitted requests 1\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:58 INFO - Done processing buckets in 0.01 min\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:58 INFO - creating document snapshots\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:59 INFO - document snapshots created\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:39:59 INFO - Completed 0 files (0.0%)  in 0.0 min. Waiting for completion\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:40:02 INFO - Completed processing 2 files in 0.064 min\n",
            "\u001b[36m(orchestrate pid=508980)\u001b[0m 00:40:02 INFO - done flushing in 0.001 sec\n",
            "00:40:12 INFO - Completed execution in 0.541 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:5 completed successfully\n",
            "CPU times: user 600 ms, sys: 375 ms, total: 974 ms\n",
            "Wall time: 33.8 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "import os\n",
        "import sys\n",
        "\n",
        "from data_processing.utils import ParamsUtils\n",
        "from fdedup_transform_ray import FdedupRayTransformConfiguration\n",
        "from data_processing_ray.runtime.ray import RayTransformLauncher\n",
        "\n",
        "# create parameters\n",
        "\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "worker_options = {\"num_cpus\" : MY_CONFIG.RAY_NUM_CPUS}\n",
        "code_location = {\"github\": \"github\", \"commit_hash\": \"12345\", \"path\": \"path\"}\n",
        "params = {\n",
        "    # where to run\n",
        "    \"run_locally\": True,\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    # Orchestration parameters\n",
        "    \"runtime_worker_options\": ParamsUtils.convert_to_ast(worker_options),\n",
        "    \"runtime_num_workers\": MY_CONFIG.RAY_RUNTIME_WORKERS,\n",
        "    # columns used\n",
        "    \"fdedup_doc_column\": \"contents\",\n",
        "    \"fdedup_id_column\": \"chunk_id\",\n",
        "    \"fdedup_cluster_column\": \"chunk_hash\",\n",
        "    # infrastructure\n",
        "    \"fdedup_bucket_cpu\": 0.3,\n",
        "    \"fdedup_doc_cpu\": 0.3,\n",
        "    \"fdedup_mhash_cpu\": 0.3,\n",
        "    \"fdedup_num_doc_actors\": 1,\n",
        "    \"fdedup_num_bucket_actors\": 1,\n",
        "    \"fdedup_num_minhash_actors\": 1,\n",
        "    \"fdedup_num_preprocessors\": 1,\n",
        "    # fuzzy parameters\n",
        "    \"fdedup_num_permutations\": 64,\n",
        "    \"fdedup_threshold\": 0.7, # (default 0.8)\n",
        "    \"fdedup_shingles_size\": 5,\n",
        "    \"fdedup_delimiters\": \" \"\n",
        "}\n",
        "\n",
        "# Pass commandline params\n",
        "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
        "\n",
        "# launch\n",
        "\n",
        "launcher = RayTransformLauncher(FdedupRayTransformConfiguration())\n",
        "\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Ray job failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a6f8cd11",
      "metadata": {
        "id": "a6f8cd11"
      },
      "source": [
        "### 8.3 - Inspect Generated output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "id": "e899ad60",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222
        },
        "id": "e899ad60",
        "outputId": "17aaaea8-a106-4c9a-ceb3-6760d92f8b59"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input data dimensions (rows x columns)=  (7, 19)\n",
            "Output data dimensions (rows x columns)=  (6, 19)\n",
            "Duplicate chunks removed  by fuzzy-dedupe:   1\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "      <th>source_document_id</th>\n",
              "      <th>contents</th>\n",
              "      <th>doc_jsonpath</th>\n",
              "      <th>page_number</th>\n",
              "      <th>bbox</th>\n",
              "      <th>document_id</th>\n",
              "      <th>chunk_id</th>\n",
              "      <th>removed</th>\n",
              "      <th>chunk_hash</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>6</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>0</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "      <td>$.main-text[3]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.20942688, 570.81555176, 375.57919312, 581...</td>\n",
              "      <td>d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...</td>\n",
              "      <td>1</td>\n",
              "      <td>[]</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.91053772, 512.46295166, 477.84887695, 534...</td>\n",
              "      <td>7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...</td>\n",
              "      <td>2</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
              "0   mars.pdf          1           0                11  pdf   \n",
              "1   mars.pdf          1           0                11  pdf   \n",
              "2  earth.pdf          1           0                11  pdf   \n",
              "3  earth.pdf          1           0                11  pdf   \n",
              "4  earth.pdf          1           0                11  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "2  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "3  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "4  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \\\n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "1  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "2  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "3  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "4  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "\n",
              "                     source_document_id  \\\n",
              "0  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "1  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "2  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "3  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "4  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "\n",
              "                                            contents    doc_jsonpath  \\\n",
              "0  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
              "1  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
              "2  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "3  Solar System\\nFor more details about our Solar...  $.main-text[3]   \n",
              "4  Earth\\nEarth is the third planet from the Sun....  $.main-text[5]   \n",
              "\n",
              "   page_number                                               bbox  \\\n",
              "0            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
              "1            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
              "2            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
              "3            1  [133.20942688, 570.81555176, 375.57919312, 581...   \n",
              "4            1  [132.91053772, 512.46295166, 477.84887695, 534...   \n",
              "\n",
              "                                         document_id  chunk_id removed  \\\n",
              "0  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6      []   \n",
              "1  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7      []   \n",
              "2  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0      []   \n",
              "3  d7be13d7dee96cf2384072d0eb01981e0e75eec2e7bc6d...         1      []   \n",
              "4  7c4a750e2215f231803a6f8078bde1e9699034fb033dd3...         2      []   \n",
              "\n",
              "   chunk_hash  \n",
              "0          -1  \n",
              "1          -1  \n",
              "2          -1  \n",
              "3           5  \n",
              "4          -1  "
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
        "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
        "print (\"Duplicate chunks removed  by fuzzy-dedupe:  \", (input_df.shape[0] - output_df.shape[0]))\n",
        "\n",
        "output_df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "id": "ab7ea52b",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "ab7ea52b",
        "outputId": "8e57385f-c925-4ac7-9e0d-ebc64e92530a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>contents</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Solar System\\nFor more details about our Solar...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nEarth is the third planet from the Sun....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>Earth\\nBasic facts about Earth:\\n· Distance fr...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename                                           contents\n",
              "0   mars.pdf  Mars\\nMars, the fourth planet from the Sun, is...\n",
              "1   mars.pdf  Basic facts about Mars:\\n· Distance from the S...\n",
              "2  earth.pdf  Solar System\\nOur solar system is a vast and f...\n",
              "3  earth.pdf  Solar System\\nFor more details about our Solar...\n",
              "4  earth.pdf  Earth\\nEarth is the third planet from the Sun....\n",
              "5  earth.pdf  Earth\\nBasic facts about Earth:\\n· Distance fr..."
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output_df[['filename', 'contents']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "id": "6bdd3515",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6bdd3515",
        "outputId": "00705442-b6ae-4238-b0f5-c94de690ecb4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "========== mars.pdf ===========\n",
            "-------Chunk 0------\n",
            "Mars\n",
            "Mars, the fourth planet from the Sun, is a cold, desert world with a thin atmosphere composed primarily of carbon dioxide. Its reddish hue comes from iron oxide, or rust, prevalent on its surface.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Basic facts about Mars:\n",
            "· Distance from the Sun: Average of 228 million kilometers (142 million miles)\n",
            "· Rotation Period: 24.6 hours (one Martian day - called a \"sol\")\n",
            "· Moons: Two small moons, Phobos and Deimos.\n",
            "-------\n",
            "========== earth.pdf ===========\n",
            "-------Chunk 0------\n",
            "Solar System\n",
            "Our solar system is a vast and fascinating expanse, comprising eight planets, five dwarf planets, numerous moons, asteroids, comets, and other celestial bodies. At its center lies the star we call the Sun.\n",
            "-------\n",
            "-------Chunk 1------\n",
            "Solar System\n",
            "For more details about our Solar system see Chapter 1.\n",
            "-------\n",
            "-------Chunk 2------\n",
            "Earth\n",
            "Earth is the third planet from the Sun. It's our home planet. Earth is the only place we know of with life.\n",
            "-------\n",
            "-------Chunk 3------\n",
            "Earth\n",
            "Basic facts about Earth:\n",
            "· Distance from the Sun: Average of 149.6 million kilometers (93 million miles)\n",
            "· Rotation Period: 24 hours (one day)\n",
            "· Moons: One moon, called Luna or simply \"the Moon\".\n",
            "-------\n"
          ]
        }
      ],
      "source": [
        "for f in output_df['filename'].unique():\n",
        "    print ('==========' , f, '===========')\n",
        "    chunks = output_df[output_df['filename'] == f]['contents']\n",
        "    for idx , chunk in enumerate(chunks):\n",
        "        print (f'-------Chunk {idx}------\\n{chunk}\\n-------')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2b34d9c6",
      "metadata": {
        "id": "2b34d9c6"
      },
      "source": [
        "### 8.4- Understanding the output\n",
        "\n",
        "So we started with 7 rows and ended up with 6.  Fuzzy dedupe removed the following **very similar** chunk.\n",
        "\n",
        "These are pretty similar chunks except for the words 'the' and 'our'\n",
        "\n",
        "**earth.pdf**\n",
        "\n",
        "`For more details about *our* Solar system see Chapter 1.`\n",
        "\n",
        "**mars.pdf**\n",
        "\n",
        "`For more details about *the* Solar system see Chapter 1.`\n",
        "\n",
        "Pretty neat, eh? 👏\n",
        "\n",
        "### Configuring Fuzzy de-dupe\n",
        "\n",
        "You can tweak fuzzy dedupe by tweaking the following parameters\n",
        "\n",
        "```python\n",
        "# fuzzy parameters\n",
        "    \"fdedup_num_permutations\": 64,\n",
        "    \"fdedup_threshold\": 0.7, #  (default 0.8)\n",
        "    \"fdedup_shingles_size\": 5,\n",
        "    \"fdedup_delimiters\": \" \"\n",
        "```\n",
        "\n",
        "In our case, we set `fdedup_threshold` parameter to 0.7.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5370950a-2a3a-4143-8218-f9b4808099ba",
      "metadata": {
        "id": "5370950a-2a3a-4143-8218-f9b4808099ba"
      },
      "source": [
        "## Step-9:   Text encoding\n",
        "\n",
        "Encode text for the vector storage."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "85aba685",
      "metadata": {
        "id": "85aba685"
      },
      "source": [
        "### 9.1 - Set Input/output Folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "id": "20a153fa-fd56-401e-86be-4f7617affcc8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "20a153fa-fd56-401e-86be-4f7617affcc8",
        "outputId": "e1795167-9fac-4b7c-9417-f655c30848a1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🏃🏼 STAGE-6: Processing input='output/05_fuzzy_dedupe_out' --> output='output/06_embeddings_out'\n"
          ]
        }
      ],
      "source": [
        "STAGE  = 6\n",
        "\n",
        "input_folder = output_fuzzy_dedupe_dir # previous output folder is the input folder for the current stage\n",
        "output_folder =  output_embeddings_dir\n",
        "\n",
        "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
        "\n",
        "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c97545f4",
      "metadata": {
        "id": "c97545f4"
      },
      "source": [
        "### 9.2 - Execute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "id": "228df6b2-bc62-494b-9697-03ece98d7853",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "228df6b2-bc62-494b-9697-03ece98d7853",
        "outputId": "f4c2cba4-aed0-4eee-873b-d1a8abf60cbd"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "00:40:14 INFO - text_encoder parameters are : {'content_column_name': 'contents', 'output_embeddings_column_name': 'embeddings', 'model_name': 'sentence-transformers/all-MiniLM-L6-v2'}\n",
            "00:40:14 INFO - pipeline id pipeline_id\n",
            "00:40:14 INFO - code location None\n",
            "00:40:14 INFO - data factory data_ is using local data access: input_folder - output/05_fuzzy_dedupe_out output_folder - output/06_embeddings_out\n",
            "00:40:14 INFO - data factory data_ max_files -1, n_sample -1\n",
            "00:40:14 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
            "00:40:14 INFO - orchestrator text_encoder started at 2024-10-20 00:40:14\n",
            "00:40:14 INFO - Number of files is 2, source profile {'max_file_size': 0.010044097900390625, 'min_file_size': 0.009777069091796875, 'total_file_size': 0.0198211669921875}\n",
            "00:40:16 INFO - Completed 1 files (50.0%) in 0.002 min\n",
            "00:40:16 INFO - Completed 2 files (100.0%) in 0.003 min\n",
            "00:40:16 INFO - Done processing 2 files, waiting for flush() completion.\n",
            "00:40:16 INFO - done flushing in 0.0 sec\n",
            "00:40:16 INFO - Completed execution in 0.024 min, execution result 0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Stage:6 completed successfully\n",
            "CPU times: user 739 ms, sys: 192 ms, total: 932 ms\n",
            "Wall time: 1.85 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
        "from text_encoder_local_python import TextEncoderPythonTransformConfiguration\n",
        "\n",
        "local_conf = {\n",
        "    \"input_folder\": input_folder,\n",
        "    \"output_folder\": output_folder,\n",
        "}\n",
        "params = {\n",
        "    # Data access. Only required parameters are specified\n",
        "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
        "    # text_encoder\n",
        "    \"text_encoder_model_name\": MY_CONFIG.EMBEDDING_MODEL,\n",
        "}\n",
        "\n",
        "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
        "# create launcher\n",
        "launcher = PythonTransformLauncher(TextEncoderPythonTransformConfiguration())\n",
        "\n",
        "return_code = launcher.launch()\n",
        "\n",
        "if return_code == 0:\n",
        "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
        "else:\n",
        "    raise Exception (\"❌ Job failed\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b734852c",
      "metadata": {
        "id": "b734852c"
      },
      "source": [
        "### 9.3 - Inspect Generated output\n",
        "\n",
        "You will see a column called `embeddings` added at the end.  This the text content converted into vectors or embeddings.  We used the model `sentence-transformers/all-MiniLM-L6-v2`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "id": "7b1c1d09",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "7b1c1d09",
        "outputId": "86c49244-9f9f-4116-fb17-c27ff6c29bc7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input data dimensions (rows x columns)=  (6, 19)\n",
            "Output data dimensions (rows x columns)=  (6, 20)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>filename</th>\n",
              "      <th>num_pages</th>\n",
              "      <th>num_tables</th>\n",
              "      <th>num_doc_elements</th>\n",
              "      <th>ext</th>\n",
              "      <th>hash</th>\n",
              "      <th>size</th>\n",
              "      <th>date_acquired</th>\n",
              "      <th>pdf_convert_time</th>\n",
              "      <th>source_filename</th>\n",
              "      <th>source_document_id</th>\n",
              "      <th>contents</th>\n",
              "      <th>doc_jsonpath</th>\n",
              "      <th>page_number</th>\n",
              "      <th>bbox</th>\n",
              "      <th>document_id</th>\n",
              "      <th>chunk_id</th>\n",
              "      <th>removed</th>\n",
              "      <th>chunk_hash</th>\n",
              "      <th>embeddings</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Mars\\nMars, the fourth planet from the Sun, is...</td>\n",
              "      <td>$.main-text[5]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87440491, 500.84011841, 477.48345947, 534...</td>\n",
              "      <td>a31663e06fac41470ecc459f5a58658a3f9997d7801053...</td>\n",
              "      <td>6</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "      <td>[0.07728295, 0.024970993, -0.043180738, 0.0580...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...</td>\n",
              "      <td>2800</td>\n",
              "      <td>2024-10-20T00:39:38.879066</td>\n",
              "      <td>0.807967</td>\n",
              "      <td>mars.pdf</td>\n",
              "      <td>bcca2c02-29d9-4226-b8b2-92a41704ab6b</td>\n",
              "      <td>Basic facts about Mars:\\n· Distance from the S...</td>\n",
              "      <td>$.main-text[6]</td>\n",
              "      <td>1</td>\n",
              "      <td>[133.2026062, 482.90710449, 237.04431152, 493....</td>\n",
              "      <td>7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...</td>\n",
              "      <td>7</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "      <td>[0.10598018, 0.025460618, 0.023627337, 0.03905...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>pdf</td>\n",
              "      <td>18713f970989055625bef22209b6f4b6830b9ca22046bf...</td>\n",
              "      <td>2686</td>\n",
              "      <td>2024-10-20T00:39:38.068313</td>\n",
              "      <td>0.820342</td>\n",
              "      <td>earth.pdf</td>\n",
              "      <td>7249c642-01d5-431d-8c42-0f2845ea6060</td>\n",
              "      <td>Solar System\\nOur solar system is a vast and f...</td>\n",
              "      <td>$.main-text[2]</td>\n",
              "      <td>1</td>\n",
              "      <td>[132.87112427, 588.96014404, 479.40917969, 623...</td>\n",
              "      <td>44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...</td>\n",
              "      <td>0</td>\n",
              "      <td>[]</td>\n",
              "      <td>-1</td>\n",
              "      <td>[0.0077404436, -0.02055944, 0.026426593, 0.011...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
              "0   mars.pdf          1           0                11  pdf   \n",
              "1   mars.pdf          1           0                11  pdf   \n",
              "2  earth.pdf          1           0                11  pdf   \n",
              "\n",
              "                                                hash  size  \\\n",
              "0  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "1  8edd5dfbf888777120b528a5d8998f2757d006df0eaef7...  2800   \n",
              "2  18713f970989055625bef22209b6f4b6830b9ca22046bf...  2686   \n",
              "\n",
              "                date_acquired  pdf_convert_time source_filename  \\\n",
              "0  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "1  2024-10-20T00:39:38.879066          0.807967        mars.pdf   \n",
              "2  2024-10-20T00:39:38.068313          0.820342       earth.pdf   \n",
              "\n",
              "                     source_document_id  \\\n",
              "0  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "1  bcca2c02-29d9-4226-b8b2-92a41704ab6b   \n",
              "2  7249c642-01d5-431d-8c42-0f2845ea6060   \n",
              "\n",
              "                                            contents    doc_jsonpath  \\\n",
              "0  Mars\\nMars, the fourth planet from the Sun, is...  $.main-text[5]   \n",
              "1  Basic facts about Mars:\\n· Distance from the S...  $.main-text[6]   \n",
              "2  Solar System\\nOur solar system is a vast and f...  $.main-text[2]   \n",
              "\n",
              "   page_number                                               bbox  \\\n",
              "0            1  [132.87440491, 500.84011841, 477.48345947, 534...   \n",
              "1            1  [133.2026062, 482.90710449, 237.04431152, 493....   \n",
              "2            1  [132.87112427, 588.96014404, 479.40917969, 623...   \n",
              "\n",
              "                                         document_id  chunk_id removed  \\\n",
              "0  a31663e06fac41470ecc459f5a58658a3f9997d7801053...         6      []   \n",
              "1  7ff317954ec5f3b15607c053c30c2b0db0f6b64cc3295a...         7      []   \n",
              "2  44c6e373258c7cdc03f75a8e96a9b160f9aa4e4baf5674...         0      []   \n",
              "\n",
              "   chunk_hash                                         embeddings  \n",
              "0          -1  [0.07728295, 0.024970993, -0.043180738, 0.0580...  \n",
              "1          -1  [0.10598018, 0.025460618, 0.023627337, 0.03905...  \n",
              "2          -1  [0.0077404436, -0.02055944, 0.026426593, 0.011...  "
            ]
          },
          "execution_count": 35,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from utils import read_parquet_files_as_df\n",
        "\n",
        "output_df = read_parquet_files_as_df(output_folder)\n",
        "\n",
        "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
        "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
        "\n",
        "output_df.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f5e12630-be6b-4188-a925-77117155617b",
      "metadata": {
        "id": "f5e12630-be6b-4188-a925-77117155617b"
      },
      "source": [
        "## Step-10: Copy output to final output dir"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
        "outputId": "aa667c65-8421-4d4d-f57e-47ccc4ea41ad"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ Copied output from 'output/06_embeddings_out' --> 'output/output_final'\n"
          ]
        }
      ],
      "source": [
        "import shutil\n",
        "\n",
        "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER_FINAL, ignore_errors=True)\n",
        "shutil.copytree(src=output_folder, dst=MY_CONFIG.OUTPUT_FOLDER_FINAL)\n",
        "\n",
        "print (f\"✅ Copied output from '{output_folder}' --> '{MY_CONFIG.OUTPUT_FOLDER_FINAL}'\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "data-prep-kit-examples-021",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0a1ed94698ca4e4291c553929e0ca66c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2eea7bc810e54eaeb325136352b71e66": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3077f04af3a9447ab98717bd3131cd8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4f63bfad92b64e7bae18e720376d402d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_709685da1c6c4164bed658357a2191bf",
            "max": 7,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0a1ed94698ca4e4291c553929e0ca66c",
            "value": 7
          }
        },
        "5dbc6889a9c243c5a922f8cc5f1a704c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6957a659451b46dab702c1c62fa9cdd2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5dbc6889a9c243c5a922f8cc5f1a704c",
            "placeholder": "​",
            "style": "IPY_MODEL_d6e520e4da004c818031ccfcc3588e5d",
            "value": " 7/7 [00:00&lt;00:00, 221.60it/s]"
          }
        },
        "709685da1c6c4164bed658357a2191bf": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7616f1b493e1461c9fd1319fae3bc10b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ebc626c0750c470db6789b26acf15f60",
            "placeholder": "​",
            "style": "IPY_MODEL_3077f04af3a9447ab98717bd3131cd8f",
            "value": "Fetching 7 files: 100%"
          }
        },
        "8226b2522ce446f6bd3a36c4e227370c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7616f1b493e1461c9fd1319fae3bc10b",
              "IPY_MODEL_4f63bfad92b64e7bae18e720376d402d",
              "IPY_MODEL_6957a659451b46dab702c1c62fa9cdd2"
            ],
            "layout": "IPY_MODEL_2eea7bc810e54eaeb325136352b71e66"
          }
        },
        "d6e520e4da004c818031ccfcc3588e5d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ebc626c0750c470db6789b26acf15f60": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}