Add guide for external services (#12)

* Add content for the indicators section

Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>

* Add guide for setting up services

Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>

---------

Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>

docs/_toc.yml

@@ -6,6 +6,7 @@ parts:
     chapters:
       - file: notebooks/examples/getting-started/setting-up-the-environment.ipynb
       - file: notebooks/examples/getting-started/openai-api.ipynb
+      - file: notebooks/examples/getting-started/setting-up-services.ipynb
       - file: notebooks/examples/getting-started/setting-up-database.ipynb
         sections:
         - file: notebooks/examples/getting-started/setting-up-database-sqlite.ipynb

notebooks/examples/getting-started/setting-up-services.ipynb

@@ -0,0 +1,142 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Setting up services"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Some of the functionalities supported by LLM4Data require external services. For example, databases or vector indices.\n",
+    "\n",
+    "In this guide, we will provide you with useful steps to get started with setting up these services so you can get started with applying LLM4Data with your own knowledge sources."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Database\n",
+    "\n",
+    "The LLM4Data library allows you to access WDI indicators using natural language by generating SQL queries in real time.\n",
+    "\n",
+    "The library uses SQL engines to store the WDI indicators to your system. You can use the lightweight SQLite library when experimenting, however, it is useful to use a more capable database service for system-critical use cases.\n",
+    "\n",
+    "We leverage the SQLAlchemy ORM to seamlessly interface with the various SQL engines. We use docker images to easily spin-up the services required in this guide."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### PostgreSQL\n",
+    "\n",
+    "Instead of the SQLite database, you can use a PostgreSQL server for storing indicator data, e.g., the WDI.\n",
+    "\n",
+    "Simply run the command below to start a docker container running the official PostgreSQL image. Set the `POSTGRES_PASSWORD` and make sure to update the `.env` file.\n",
+    "\n",
+    "```\n",
+    "docker run --name postgres -e POSTGRES_PASSWORD=<password> -d postgres\n",
+    "```\n",
+    "\n",
+    "Here's the relevant variables in the `.env` file that must be updated. Note the `<password>` must be replaced with the value you set in the `POSTGRES_PASSWORD` variable when starting the docker container.\n",
+    "\n",
+    "```## POSTRESQL\n",
+    "WDI_DB_ENGINE=\"postgresql\"\n",
+    "WDI_DB_HOST=\"localhost\"\n",
+    "WDI_DB_USERNAME=\"postgres\"\n",
+    "WDI_DB_PASSWORD=<password>\n",
+    "WDI_DB_PORT=5432\n",
+    "```\n",
+    "\n",
+    "Please check the [official page](https://hub.docker.com/_/postgres) for updates."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Vector Index\n",
+    "\n",
+    "We use vector embeddings to perform semantic search of relevant texts to prompts. The vectors for a collection of texts must be stored for retrieval. In simple applications, the vectors can be in-memory which does not require any external services. However, in more complex applications and also to save time in generating the embeddings for a large collection of texts, we can use a vector index.\n",
+    "\n",
+    "A vector index stores pre-computed vector embeddings for a collection og texts. This saves time by not requiring repeated processing of texts to generate these embeddings. Also, most vector indices implement efficient approximate retrieval algorithms that make searching for millions or billions of vectors fast."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Qdrant\n",
+    "\n",
+    "The LLM4Data implement support for Qdrant for indexing of vector embeddings.\n",
+    "\n",
+    "Qdrant can be easily set up using its official docker image. \n",
+    "\n",
+    "```\n",
+    "docker run -p 6333:6333 qdrant/qdrant\n",
+    "```\n",
+    "\n",
+    "Please check the [official page](https://hub.docker.com/r/qdrant/qdrant) for updates."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Monitoring Qdrant\n",
+    "\n",
+    "Qdrant has implemented a lightweight interface to show to monitor the status of the Qdrant service. The Qdrant UI is available at the following URL:\n",
+    "\n",
+    "```bash\n",
+    "https://github.com/qdrant/qdrant-web-ui\n",
+    "```\n",
+    "\n",
+    "Clone the repo, install serve and vite, and run in development mode.\n",
+    "\n",
+    "```bash\n",
+    "git clone https://github.com/qdrant/qdrant-web-ui.git\n",
+    "cd qdrant-web-ui\n",
+    "\n",
+    "npm install -g serve\n",
+    "npm install vite\n",
+    "\n",
+    "npm start\n",
+    "```\n",
+    "\n",
+    "You will then be able to see stats on the Qdrant service."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "After following this guide, you should be able to start storing the indicators data into a PostgreSQL database. Should also be able to start inserting vector embeddings into a Qdrant vector index.\n",
+    "\n",
+    "Watch out for future guides to get started with different databases or vector indices as we introduce more features into LLM4Data!"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

notebooks/examples/indicators/README.md

@@ -1 +1,7 @@
 # Indicators
+
+This section showcases some examples of how LLM4Data can be used to interact with development indicators.
+
+The first few examples will focus on the World Development Indicators (WDI).
+
+Other indicators will be added in the future.