Documentation for PostgreSQL integration testing

CHANGELOG.md

@@ -8,7 +8,9 @@
 * [Doc] Document --persist-replace in API section (#539)
 * [Fix] Fixed CI issue by updating `invalid_connection_string_duckdb` in `test_magic.py` (#631)
 * [Fix] Refactored `ResultSet` to lazy loading (#470)
+* [Fix] Fixed pytest command getting stuck while running PostgreSQL integration tests in absence of psycopg2 connector library ([#643](https://github.com/ploomber/jupysql/issues/643))
 * [Fix] Removed `WITH` when a snippet does not have a dependency (#657)
+* [Doc] Added guide on running PostgreSQL integration tests ([#643](https://github.com/ploomber/jupysql/issues/643))
 
 ## 0.7.9 (2023-06-19)
 

doc/_toc.yml

@@ -37,6 +37,7 @@ parts:
       - file: integrations/questdb
       - file: integrations/oracle
       - file: integrations/trinodb
+      - file: integrations/postgres-test-doc
 
 
   - caption: API Reference

doc/community/developer-guide.md

@@ -353,6 +353,9 @@ To run some of the tests:
 pytest src/tests/integration/test_generic_db_operations.py::test_profile_query
 ```
 
+### PostgreSQL Integration Tests
+To run the tests for PostgreSQL DB integration follow the guide [here](../integrations/postgres-test-doc.ipynb)
+
 ### Integration tests with cloud databases
 
 We run integration tests against cloud databases like Snowflake, which requires using pre-registered accounts to evaluate their behavior. To initiate these tests, please create a branch in our [ploomber/jupyter repository](https://github.com/ploomber/jupysql).

doc/conf.py

@@ -26,6 +26,7 @@
     "integrations/trinodb.ipynb",
     "integrations/oracle.ipynb",
     "integrations/snowflake.ipynb",
+    "integrations/postgres-test-doc.ipynb"
 ]
 nb_execution_in_temp = True
 nb_execution_show_tb = True

doc/howto/postgres-install.md

@@ -25,6 +25,6 @@ Ensure that you are using `pgspecial 1.x`. `pgspecial 2.x` has migrated to `psyc
 ```sh
 conda install "pgspecial<2" -c conda-forge
 ```
- 
+
 
 If you have trouble getting it to work, [message us on Slack.](https://ploomber.io/community)

doc/integrations/postgres-test-doc.ipynb

@@ -0,0 +1,312 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "0e5d0f1c",
+   "metadata": {},
+   "source": [
+    "# PostgreSQL Integration Test Guide\n",
+    "\n",
+    "This guide will help you with running Integration Tests for PostgreSQL with JupySQL"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ac901820",
+   "metadata": {},
+   "source": [
+    "To run this tutorial you will need to start with setting up a local development environment for JupySQL. "
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "565f8648",
+   "metadata": {},
+   "source": [
+    "## Step 1 \n",
+    "To setup a local development environment follow setup guide over here :\n",
+    "[Setup Guide](https://ploomber-contributing.readthedocs.io/en/latest/contributing/setup.html)\n",
+    "\n",
+    "Use [jupysql](https://github.com/ploomber/jupysql) repository to setup the dev environment for performing the tests in this guide"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ddf54361",
+   "metadata": {},
+   "source": [
+    "## Step 2\n",
+    "Once the local environment is setup for JupySQL, activate the environment by running"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "98bc699a",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "conda activate jupysql\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "88a05f93",
+   "metadata": {},
+   "source": [
+    "## Step 3\n",
+    "\n",
+    "For testing PostgreSQL Integration we will need to install following dependencies "
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "e5a346bc",
+   "metadata": {},
+   "source": [
+    "### Install Dependencies\n",
+    "\n",
+    "To connect to a PostgreSQL database from Python, you need a client library. We recommend using `psycopg2`, but there are others like `pg8000`, and `asyncpg`. JupySQL supports the [following connectors.](https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#dialect-postgresql)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "9fb18c1e",
+   "metadata": {},
+   "source": [
+    "#### Installing PostgreSQL client `psycopg2`\n",
+    "If you have `conda` installed, it is more reliable to use it:"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ee76468e",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "conda install psycopg2 -c conda-forge\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "7f132c2c",
+   "metadata": {},
+   "source": [
+    "#### Installing `pgspecial`\n",
+    "\n",
+    "Ensure that you are using `pgspecial 1.x`. `pgspecial 2.x` has migrated to `psycopg3` and thus does not yield informative error messages."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "c1a17ab5",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "conda install \"pgspecial<2\" -c conda-forge\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "b3867555",
+   "metadata": {},
+   "source": [
+    "#### Installing dockerctx\n",
+    "\n",
+    "dockerctx is a context manager for managing the lifetime of a docker container. We'll be using this library as we're spinning up docker container to launch the database instance and running integration tests over it.\n",
+    "\n",
+    "\n",
+    "```{tip}\n",
+    "Make sure the earlier dependencies - psycopg2, pgspecial are installed before installing dockerctx, otherwise as it dependes upon those libraries it might throw an error\n",
+    "```\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "163b77ce",
+   "metadata": {},
+   "source": [
+    "```sh \n",
+    "pip install dockerctx"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "2ab6a626",
+   "metadata": {},
+   "source": [
+    "If you have trouble getting it to work, [message us on Slack.](https://ploomber.io/community)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "cc869fb9",
+   "metadata": {},
+   "source": [
+    "To get PostgreSQL instance up and running, you need to install following Python packages:"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "b7ce9b76",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "```sh\n",
+    "pip install jupysql pandas pyarrow --quiet"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "5e246dc2",
+   "metadata": {},
+   "source": [
+    "```{tip}\n",
+    "If you have issues, check out our [installation guide](../howto/postgres-install.md) or [message us on Slack.](https://ploomber.io/community)\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "5586988f",
+   "metadata": {},
+   "source": [
+    "```{Important}\n",
+    "You need to make sure that [Docker Desktop](https://docs.docker.com/desktop/) is installed and running before moving forward as tests are configured to spin up new PostgreSQL container, run defined tests on it, and clean up.\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "cae73725",
+   "metadata": {},
+   "source": [
+    "### Running Integration Tests\n",
+    "\n",
+    "Now, as we have all the dependencies installed we are all set to run the integration tests. Integrations tests are located at ```\"src/tests/integration/\"``` we will run them using following commands."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "a802aa91",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "pytest src/tests/integration/test_postgreSQL.py\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ad7a909d",
+   "metadata": {},
+   "source": [
+    "If everything goes well, you'll see the tests Passed as the output after running the command"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3b843866",
+   "metadata": {},
+   "source": [
+    "The other set of tests which test for generic db operations are located in ```\"src/tests/integration/test_generic_db_operations.py\"```. We will run them for our PostgreSQL DB by running following command"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "e0154e8f",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "pytest src/tests/integration/test_generic_db_operations.py -k \"ip_with_postgreSQL\"\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3048edc5",
+   "metadata": {},
+   "source": [
+    "```\"-k\"``` flag in above command helps us to filter and run the tests which are only specific to PostgreSQL DB integration"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "d375198a",
+   "metadata": {},
+   "source": [
+    "### Common Errors"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "0908a38e",
+   "metadata": {},
+   "source": [
+    "```{Important}\n",
+    "The following error might show up if docker is not installed and/or the container is not properly running. Hence keep an eye that you see the newly spun DB container while running the tests.\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "681d0f6f",
+   "metadata": {},
+   "source": [
+    "```sh\n",
+    "docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

src/sql/_testing.py

@@ -228,7 +228,7 @@ def postgres(is_bypass_init=False):
                 "POSTGRES_USER": db_config["username"],
                 "POSTGRES_PASSWORD": db_config["password"],
             },
-            ready_test=lambda: database_ready(database="postgreSQL"),
+            ready_test=lambda: database_ready(database="postgreSQL", timeout=5), ## Passing timeout parameter for PostgreSQL to exit quickly in case of absense of connector library (psycopg2)
             healthcheck={
                 "test": "pg_isready",
                 "interval": 10000000000,