Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref: use new prepare code and remove some old #778

Merged
merged 7 commits into from
Sep 27, 2023

Conversation

jordanrfrazier
Copy link
Collaborator

This removes the column_behavior in favor of the simpler, newer prepare_batch logic. This also fixes the "floats not allowed as timestamps" behavior, since the new logic handles that.

Closes #776

@cla-bot cla-bot bot added the cla-signed Set when all authors of a PR have signed our CLA label Sep 26, 2023
@bjchambers
Copy link
Collaborator

If this passes all the tests... that's awesome. Deletes a bunch of old code and fixes bugs. I think the one thing that we maybe lose is the ability to drop/evolve columns and such -- but I think that can be added to the existing prepare batch -- if we know the "expected" schema, we can just omit/evolve columns based on the file schema and expected schema. Definitely simpler!

@jordanrfrazier jordanrfrazier added this pull request to the merge queue Sep 27, 2023
Merged via the queue into main with commit 72c03a5 Sep 27, 2023
32 checks passed
@jordanrfrazier jordanrfrazier deleted the 776-use-prepare-batch-over-column-behavior branch September 27, 2023 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed Set when all authors of a PR have signed our CLA refactor sparrow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Loading parquet sources with Float timestamp is broken
2 participants