Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate ephemeral database(s) with CI for read/write testing #12815

Open
alexander-beedie opened this issue Nov 30, 2023 · 0 comments
Open
Labels
A-io-database Area: reading/writing to databases A-sql Area: Polars SQL functionality accepted Ready for implementation ci Related to the continuous integration setup enhancement New feature or an improvement of an existing feature internal An internal refactor or improvement

Comments

@alexander-beedie
Copy link
Collaborator

alexander-beedie commented Nov 30, 2023

Description

@stinodego: As promised ;)

Currently read_database and write_database unit tests rely on SQLite when running on CI, which is insufficient to provide quality testing due to the numerous differences between that and something like PostgreSQL, MSSQL, Redshift, Snowflake, etc. While extending these functions I've had to rely on validating new functionality against a containerised PostgreSQL (and a few other DBs) running off a NAS at home, or using Polars at work where we have data in (amongst others) Redshift, Databricks, and BigQuery stores.

We should enhance CI (and the associated unit tests) to, at a minimum, support an ephemeral PostgreSQL test target as well. If there are other databases that support simple containerisation (or some equivalent facility) then we could think about adding more than PostgreSQL later 🤔

@alexander-beedie alexander-beedie added enhancement New feature or an improvement of an existing feature ci Related to the continuous integration setup labels Nov 30, 2023
@alexander-beedie alexander-beedie changed the title Integrate ephemeral database with CI for read/write testing Integrate ephemeral database(s) with CI for read/write testing Nov 30, 2023
@alexander-beedie alexander-beedie added the A-sql Area: Polars SQL functionality label Nov 30, 2023
@stinodego stinodego added accepted Ready for implementation internal An internal refactor or improvement labels Nov 30, 2023
@alexander-beedie alexander-beedie added the A-io-database Area: reading/writing to databases label Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io-database Area: reading/writing to databases A-sql Area: Polars SQL functionality accepted Ready for implementation ci Related to the continuous integration setup enhancement New feature or an improvement of an existing feature internal An internal refactor or improvement
Projects
Status: Ready
Development

No branches or pull requests

2 participants