Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for PostgreSQL integration testing #663

Closed
wants to merge 12 commits into from
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
* [Doc] Document --persist-replace in API section (#539)
* [Fix] Fixed CI issue by updating `invalid_connection_string_duckdb` in `test_magic.py` (#631)
* [Fix] Refactored `ResultSet` to lazy loading (#470)
* [Fix] Fixed pytest command getting stuck while running PostgreSQL integration tests in absence of psycopg2 connector library ([#643](https://github.com/ploomber/jupysql/issues/643))
* [Fix] Removed `WITH` when a snippet does not have a dependency (#657)
* [Doc] Added guide on running PostgreSQL integration tests ([#643](https://github.com/ploomber/jupysql/issues/643))

## 0.7.9 (2023-06-19)

Expand Down
1 change: 1 addition & 0 deletions doc/_toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ parts:
- file: integrations/questdb
- file: integrations/oracle
- file: integrations/trinodb
- file: integrations/postgres-test-doc


- caption: API Reference
Expand Down
3 changes: 3 additions & 0 deletions doc/community/developer-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,9 @@ To run some of the tests:
pytest src/tests/integration/test_generic_db_operations.py::test_profile_query
```

### PostgreSQL Integration Tests
To run the tests for PostgreSQL DB integration follow the guide [here](../integrations/postgres-test-doc.ipynb)

### Integration tests with cloud databases

We run integration tests against cloud databases like Snowflake, which requires using pre-registered accounts to evaluate their behavior. To initiate these tests, please create a branch in our [ploomber/jupyter repository](https://github.com/ploomber/jupysql).
Expand Down
1 change: 1 addition & 0 deletions doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"integrations/trinodb.ipynb",
"integrations/oracle.ipynb",
"integrations/snowflake.ipynb",
"integrations/postgres-test-doc.ipynb"
]
nb_execution_in_temp = True
nb_execution_show_tb = True
Expand Down
2 changes: 1 addition & 1 deletion doc/howto/postgres-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,6 @@ Ensure that you are using `pgspecial 1.x`. `pgspecial 2.x` has migrated to `psyc
```sh
conda install "pgspecial<2" -c conda-forge
```


If you have trouble getting it to work, [message us on Slack.](https://ploomber.io/community)
312 changes: 312 additions & 0 deletions doc/integrations/postgres-test-doc.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,312 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "0e5d0f1c",
"metadata": {},
"source": [
"# PostgreSQL Integration Test Guide\n",
"\n",
"This guide will help you with running Integration Tests for PostgreSQL with JupySQL"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ac901820",
"metadata": {},
"source": [
"To run this tutorial you will need to start with setting up a local development environment for JupySQL. "
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "565f8648",
"metadata": {},
"source": [
"## Step 1 \n",
"To setup a local development environment follow setup guide over here :\n",
"[Setup Guide](https://ploomber-contributing.readthedocs.io/en/latest/contributing/setup.html)\n",
"\n",
"Use [jupysql](https://github.com/ploomber/jupysql) repository to setup the dev environment for performing the tests in this guide"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ddf54361",
"metadata": {},
"source": [
"## Step 2\n",
"Once the local environment is setup for JupySQL, activate the environment by running"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "98bc699a",
"metadata": {},
"source": [
"```sh\n",
"conda activate jupysql\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "88a05f93",
"metadata": {},
"source": [
"## Step 3\n",
"\n",
"For testing PostgreSQL Integration we will need to install following dependencies "
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e5a346bc",
"metadata": {},
"source": [
"### Install Dependencies\n",
"\n",
"To connect to a PostgreSQL database from Python, you need a client library. We recommend using `psycopg2`, but there are others like `pg8000`, and `asyncpg`. JupySQL supports the [following connectors.](https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#dialect-postgresql)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9fb18c1e",
"metadata": {},
"source": [
"#### Installing PostgreSQL client `psycopg2`\n",
"If you have `conda` installed, it is more reliable to use it:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ee76468e",
"metadata": {},
"source": [
"```sh\n",
"conda install psycopg2 -c conda-forge\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7f132c2c",
"metadata": {},
"source": [
"#### Installing `pgspecial`\n",
"\n",
"Ensure that you are using `pgspecial 1.x`. `pgspecial 2.x` has migrated to `psycopg3` and thus does not yield informative error messages."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c1a17ab5",
"metadata": {},
"source": [
"```sh\n",
"conda install \"pgspecial<2\" -c conda-forge\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b3867555",
"metadata": {},
"source": [
"#### Installing dockerctx\n",
"\n",
"dockerctx is a context manager for managing the lifetime of a docker container. We'll be using this library as we're spinning up docker container to launch the database instance and running integration tests over it.\n",
"\n",
"\n",
"```{tip}\n",
"Make sure the earlier dependencies - psycopg2, pgspecial are installed before installing dockerctx, otherwise as it dependes upon those libraries it might throw an error\n",
"```\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "163b77ce",
"metadata": {},
"source": [
"```sh \n",
"pip install dockerctx"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2ab6a626",
"metadata": {},
"source": [
"If you have trouble getting it to work, [message us on Slack.](https://ploomber.io/community)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cc869fb9",
"metadata": {},
"source": [
"To get PostgreSQL instance up and running, you need to install following Python packages:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b7ce9b76",
"metadata": {
"tags": []
},
"source": [
"```sh\n",
"pip install jupysql pandas pyarrow --quiet"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5e246dc2",
"metadata": {},
"source": [
"```{tip}\n",
"If you have issues, check out our [installation guide](../howto/postgres-install.md) or [message us on Slack.](https://ploomber.io/community)\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5586988f",
"metadata": {},
"source": [
"```{Important}\n",
"You need to make sure that [Docker Desktop](https://docs.docker.com/desktop/) is installed and running before moving forward as tests are configured to spin up new PostgreSQL container, run defined tests on it, and clean up.\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cae73725",
"metadata": {},
"source": [
"### Running Integration Tests\n",
"\n",
"Now, as we have all the dependencies installed we are all set to run the integration tests. Integrations tests are located at ```\"src/tests/integration/\"``` we will run them using following commands."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "a802aa91",
"metadata": {},
"source": [
"```sh\n",
"pytest src/tests/integration/test_postgreSQL.py\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ad7a909d",
"metadata": {},
"source": [
"If everything goes well, you'll see the tests Passed as the output after running the command"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3b843866",
"metadata": {},
"source": [
"The other set of tests which test for generic db operations are located in ```\"src/tests/integration/test_generic_db_operations.py\"```. We will run them for our PostgreSQL DB by running following command"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e0154e8f",
"metadata": {},
"source": [
"```sh\n",
"pytest src/tests/integration/test_generic_db_operations.py -k \"ip_with_postgreSQL\"\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3048edc5",
"metadata": {},
"source": [
"```\"-k\"``` flag in above command helps us to filter and run the tests which are only specific to PostgreSQL DB integration"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d375198a",
"metadata": {},
"source": [
"### Common Errors"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0908a38e",
"metadata": {},
"source": [
"```{Important}\n",
"The following error might show up if docker is not installed and/or the container is not properly running. Hence keep an eye that you see the newly spun DB container while running the tests.\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "681d0f6f",
"metadata": {},
"source": [
"```sh\n",
"docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
2 changes: 1 addition & 1 deletion src/sql/_testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ def postgres(is_bypass_init=False):
"POSTGRES_USER": db_config["username"],
"POSTGRES_PASSWORD": db_config["password"],
},
ready_test=lambda: database_ready(database="postgreSQL"),
ready_test=lambda: database_ready(database="postgreSQL", timeout=5), ## Passing timeout parameter for PostgreSQL to exit quickly in case of absense of connector library (psycopg2)
healthcheck={
"test": "pg_isready",
"interval": 10000000000,
Expand Down