2 indexer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run First"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from IPython.display import Markdown\n",
    "\n",
    "def display_md(content):\n",
    "  display(Markdown(content))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Indexer Architecture\n",
    "In this course, we are indexing a small number of HTML files. However, in production situations, you're likely to encounter some situations that significantly increase the complexity of accurate indexing:  \n",
    "- PDFs contain much of the world's unstructured data\n",
    "- Parsing PDFs with vision can require layout understanding, which is not a generally solved problem\n",
    "- PDFs often contain tables, graphics, footnotes, equations, etc that require special handling\n",
    "- Many business cases require indexing highly heterogenous document layouts\n",
    "\n",
    "Since accurately indexing files is the beginning of your inference pipeline, this is often one of the most consequential engineering problems to perform well at.\n",
    "\n",
    "## loaders.py\n",
    "Look at `./workshop-code/indexer_components/loaders.py`\n",
    "\n",
    "The function here is pretty simple. It downloads a file at a specified URI, then saves it to a cache so it doesn't need to be downloaded on each subsequent run in the notebook. The preprocessor will consume this file in the next step. \n",
    "\n",
    "### Task: Read the Code\n",
    "This part isn't interesting, so just look at the code and understand what it does. If you have any questions, let one of us know.\n",
    "\n",
    "Note that the proper way to do caching is by using the HTTP response's `ETag`, `Last-Modified`, and `Cache-Control` headers, but I didn't do that here. If you want extra credit, feel free to send me a pull request with the corrected code.\n",
    "\n",
    "## preprocessors.py\n",
    "Look at `./workshop-code/indexer_components/preprocessors.py`\n",
    "\n",
    "In order to reduce inference costs, we want to strip away all of the HTML syntax besides the human-readable body text of the documents. This is a design decision. In real life, you may, for example, preserve more of the HTML markup to reain the context of how the document is structured.\n",
    "\n",
    "### Easy Task: Configure Beautiful Soup for A Simply Structured Blog Post\n",
    "\n",
    "The unprocessed HTML blog post looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workshop_code.indexer_components.loaders       import DocLoader\n",
    "\n",
    "blog_post_uri = \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
    "doc_content = DocLoader.load_html(blog_post_uri)\n",
    "display_md(doc_content[3400:3800])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The processed HTML blog post should like like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cheat_code.indexer_components.loaders       import DocLoader\n",
    "from cheat_code.indexer_components.preprocessors import GithubBlogpostPreprocessor\n",
    "\n",
    "blog_post_uri = \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
    "blog_post_html = DocLoader.load_html(blog_post_uri)\n",
    "preprocessor = GithubBlogpostPreprocessor()\n",
    "cleaned_doc_content = preprocessor.get_text(blog_post_html)\n",
    "\n",
    "display_md(cleaned_doc_content[0:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the method `GithubBlogpostPreprocessor.get_text()` in `./workshop_code/preprocessors.py` such that the `cleaned_text` looks like the above output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workshop_code.indexer_components.loaders       import DocLoader\n",
    "from workshop_code.indexer_components.preprocessors import GithubBlogpostPreprocessor\n",
    "\n",
    "blog_post_uri = \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
    "blog_post_html = DocLoader.load_html(blog_post_uri)\n",
    "preprocessor = GithubBlogpostPreprocessor()\n",
    "cleaned_doc_content = preprocessor.get_text(blog_post_html)\n",
    "\n",
    "display_md(cleaned_doc_content[0:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Realistic BeautifulSoup Task: Copy or Write a Preprocessor for the RAG Survey Paper\n",
    "I suggest just copying the cheat code. Writing this whole preprocessor will take the rest of the session.\n",
    "\n",
    "The HTML structure of the Arxiv paper is more complex than the blog post. You can try implementing some of `ArxivHtmlPaperPreprocessor` to see for yourself. But, I suggest just copying the cheat-code. \n",
    "\n",
    "Here is the working implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cheat_code.indexer_components.loaders       import DocLoader\n",
    "from cheat_code.indexer_components.preprocessors import ArxivHtmlPaperPreprocessor\n",
    "\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "rag_survey_paper_html = DocLoader.load_html(rag_survey_paper_uri)\n",
    "preprocessor = ArxivHtmlPaperPreprocessor()\n",
    "cleaned_doc_content = preprocessor.get_text(rag_survey_paper_html)\n",
    "\n",
    "display_md(cleaned_doc_content[0:2000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, here is the implementation for you to complete:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workshop_code.indexer_components.loaders       import DocLoader\n",
    "from workshop_code.indexer_components.preprocessors import ArxivHtmlPaperPreprocessor\n",
    "\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "rag_survey_paper_html = DocLoader.load_html(rag_survey_paper_uri)\n",
    "preprocessor = ArxivHtmlPaperPreprocessor()\n",
    "cleaned_doc_content = preprocessor.get_text(rag_survey_paper_html)\n",
    "\n",
    "display_md(cleaned_doc_content[0:2000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hard Task, not recommended for today: Write a Preprocessor for the PDF Version of the RAG Survey Paper\n",
    "In production applications, you're likely to need to do inference on PDFs. Today this is often a non-trivial task. The most popular open source solution is Tesseract. However, Tesseract often underperforms computer vision-based services from vendors like Google Cloud and AWS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## text_splitters.py\n",
    "Because LLM context windows are limited, semantic indexing strategies rely on text splitting. In this tutorial, we use the most naive strategy, character text splitting. To find inspiration or source code for more strategies, I look at LlamaIndex and Langchain. However, in some production situations, it will make sense to write a text splitter specific to your needs.\n",
    "\n",
    "### Text splitting task #1: examine and copy the code for the text splitter\n",
    "Here is the working implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cheat_code.indexer_components.loaders        import DocLoader\n",
    "from cheat_code.indexer_components.preprocessors  import ArxivHtmlPaperPreprocessor\n",
    "from cheat_code.indexer_components.text_splitters import SimpleCharacterTextSplitter\n",
    "\n",
    "CHUNK_SIZE = 250\n",
    "OVERLAP_SIZE = 25\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "preprocessor = ArxivHtmlPaperPreprocessor()\n",
    "text_splitter = SimpleCharacterTextSplitter(CHUNK_SIZE, OVERLAP_SIZE)\n",
    "\n",
    "rag_survey_paper_html = DocLoader.load_html(rag_survey_paper_uri)\n",
    "cleaned_doc_content = preprocessor.get_text(rag_survey_paper_html)\n",
    "text_splits = text_splitter.split_text(cleaned_doc_content)\n",
    "\n",
    "display_md(text_splits[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the code from `cheat_code/indexer_components/text_splitters.py` to `workshop_code/` so that the code below works: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workshop_code.indexer_components.loaders        import DocLoader\n",
    "from workshop_code.indexer_components.preprocessors  import ArxivHtmlPaperPreprocessor\n",
    "from workshop_code.indexer_components.text_splitters import SimpleCharacterTextSplitter\n",
    "\n",
    "CHUNK_SIZE = 250\n",
    "OVERLAP_SIZE = 25\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "preprocessor = ArxivHtmlPaperPreprocessor()\n",
    "text_splitter = SimpleCharacterTextSplitter(CHUNK_SIZE, OVERLAP_SIZE)\n",
    "\n",
    "rag_survey_paper_html = DocLoader.load_html(rag_survey_paper_uri)\n",
    "cleaned_doc_content = preprocessor.get_text(rag_survey_paper_html)\n",
    "text_splits = text_splitter.split_text(cleaned_doc_content)\n",
    "\n",
    "display_md(text_splits[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Splitting Task #2: look at alternative text splitters\n",
    "Make a mental note of the other text splitters available here:\n",
    "- [Langchain: Text Splitters](https://python.langchain.com/v0.2/docs/how_to/#text-splitters)\n",
    "- [LlamaIndex: Text Splitters](https://medium.com/@bavalpreetsinghh/llamaindex-chunking-strategies-for-large-language-models-part-1-ded1218cfd30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings: vectorizers.py\n",
    "Embeddings of text is currently the most common method of preparing human-readable text so that they can be compared to each other for relatedness. Currently, OpenAI's embedding models rank amongst the highest performing, so we use theirs.  \n",
    "\n",
    "OpenAI's text embedding models take up to 8191 tokens as input and convert them to a vector of dimension 1536 for `text-embedding-3-small` or 3072 for `text-embedding-3-large`.\n",
    "\n",
    "### Embeddings Task #1: Use OpenAI's Embeddings API\n",
    "The embeddings code for your naive RAG pipeline should behave like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cheat_code.common_components.vectorizers import Vectorizer\n",
    "\n",
    "example_text_splits = [\"Mary had a\", \"little lamb\"]\n",
    "vectorizer = Vectorizer()\n",
    "embeddings_of_example_splits = vectorizer.vectorize_text_splits(example_text_splits)\n",
    "\n",
    "rows = len(embeddings_of_example_splits)\n",
    "columns = len(embeddings_of_example_splits[0])\n",
    "\n",
    "print(f\"Dimensions: {rows}x{columns}\")\n",
    "print(embeddings_of_example_splits[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement `vectorize_text_splits()` in `workshop_code/common_components/vectorizers.py` by referencing the [OpenAI embedding API's documentation](https://platform.openai.com/docs/api-reference/embeddings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workshop_code.common_components.vectorizers import Vectorizer\n",
    "\n",
    "example_text_splits = [\"Mary had a\", \"little lamb\"]\n",
    "vectorizer = Vectorizer()\n",
    "embeddings_of_example_splits = vectorizer.vectorize_text_splits(example_text_splits)\n",
    "\n",
    "rows = len(embeddings_of_example_splits)\n",
    "columns = len(embeddings_of_example_splits[0])\n",
    "\n",
    "print(f\"Dimensions: {rows}x{columns}\")\n",
    "print(embeddings_of_example_splits[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector Database: vectordb_client_adapters.py\n",
    "In production settings, you're likely to store your vectors in a database. In this tutorial, we are using Couchbase.\n",
    "\n",
    "### Vector DB Task #1: Understand the indexer code\n",
    "Open `indexers.py` in `./workshop_code/`. Look over how the `CouchbaseClientAdapter` is used, and look at how its methods are implemented. If something doesn't make sense, ask a question.\n",
    "\n",
    "## The Complete Indexer: indexer.py\n",
    "### Indexer Task: test that your indexer works\n",
    "Your indexer should give output like the cheat_code version below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from cheat_code.indexers import NaiveIndexer\n",
    "from cheat_code.common_components.vectorizers import Vectorizer\n",
    "\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "vectorizer = Vectorizer()\n",
    "indexer = NaiveIndexer(vectorizer)\n",
    "indexer.index(rag_survey_paper_uri)\n",
    "time.sleep(5)\n",
    "num_db_entries = indexer._client_adapter.count_entries()\n",
    "print(f\"Number of text chunks in db: {num_db_entries}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run your indexer below to see if it works the same way. If it doesn't, something is broken."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from workshop_code.indexers import NaiveIndexer\n",
    "from workshop_code.common_components.vectorizers import Vectorizer\n",
    "\n",
    "rag_survey_paper_uri = \"https://arxiv.org/html/2312.10997v5\"\n",
    "vectorizer = Vectorizer()\n",
    "indexer = NaiveIndexer(vectorizer)\n",
    "indexer.index(rag_survey_paper_uri)\n",
    "time.sleep(5)\n",
    "num_db_entries = indexer._client_adapter.count_entries()\n",
    "print(f\"Number of text chunks in db: {num_db_entries}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}