Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(postgres): fix json type conversion in to_pyarrow output #8439

Merged
merged 6 commits into from
Feb 27, 2024

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Feb 24, 2024

Localize custom pyarrow json serialization to snowflake. This custom type doesn't compose well (e.g., inside structs, which aren't directly supported in snowflake anyway) and was causing problems for other backends when trying to convert rows into a struct of the table schema and then into a proper table.

Fixes #8318.

@cpcloud cpcloud added this to the 9.0 milestone Feb 24, 2024
@cpcloud cpcloud added bug Incorrect behavior inside of ibis refactor Issues or PRs related to refactoring the codebase postgres The PostgreSQL backend snowflake The Snowflake backend labels Feb 24, 2024
Copy link
Contributor

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

@cpcloud cpcloud changed the title postgres json to pyarrow fix(postgres): fix json type conversion in to_pyarrow output Feb 24, 2024
@cpcloud cpcloud force-pushed the postgres-json-to-pyarrow branch 2 times, most recently from 7009f3c to 8b0900f Compare February 24, 2024 18:33
@cpcloud cpcloud added the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 24, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 24, 2024
@cpcloud cpcloud force-pushed the postgres-json-to-pyarrow branch 3 times, most recently from 89bb386 to ae21232 Compare February 25, 2024 11:47
@cpcloud
Copy link
Member Author

cpcloud commented Feb 25, 2024

I localized the pyarrow json type to the snowflake backend to preserve the pretty printing of array/map/struct types.

@cpcloud cpcloud force-pushed the postgres-json-to-pyarrow branch from ae21232 to bc3f809 Compare February 25, 2024 12:09
@cpcloud cpcloud added the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 25, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 25, 2024
@cpcloud cpcloud added backends Issues related to all backends ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests labels Feb 25, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 25, 2024
@cpcloud cpcloud force-pushed the postgres-json-to-pyarrow branch from 7be7895 to c28e103 Compare February 25, 2024 15:42
@cpcloud cpcloud added the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 25, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Run BigQuery, Snowflake, Databricks, and Athena backend tests label Feb 25, 2024
Copy link
Member

@kszucs kszucs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kszucs kszucs merged commit b338517 into ibis-project:main Feb 27, 2024
79 checks passed
@cpcloud cpcloud deleted the postgres-json-to-pyarrow branch April 1, 2024 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backends Issues related to all backends bug Incorrect behavior inside of ibis postgres The PostgreSQL backend refactor Issues or PRs related to refactoring the codebase snowflake The Snowflake backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug(postgres): to_pyarrow fails with json type
2 participants