Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet: add variant type support #11653

Closed
wants to merge 6 commits into from

Conversation

aihuaxu
Copy link
Contributor

@aihuaxu aihuaxu commented Nov 26, 2024

This is to enable writing Variant data to Parquet file and read from Parquet file in the format of struct of metadata and data.

This PR adds:

  • A Parquet Variant implementation which currently uses Spark-Variant library as the dependency. We can reimplement inside the Iceberg-Core library to remove such dependency for now before that is moved to Parquet project.
  • Add necessary Variant support in Parquet reader/writer.

TODO:

  1. Add more test coverage.
  2. Implement Variant encoding/decoding to remove Spark-Variant library dependency.
  3. Add Parquet logical type Variant so ParquetTypeVisitor can understand Variant type from Parquet annotation.

Part of: #11178.

@aihuaxu aihuaxu force-pushed the variant-type-parquet branch from 09b7fe3 to 7679f84 Compare December 23, 2024 20:23
Copy link

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Jan 23, 2025
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant