feat(datafusion): use pyarrow for type conversion #9299
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpcloud
force-pushed
the
datafusion-simpler-types-from-query
branch
3 times, most recently
from
b5dc24b to
02f1bfc
Compare
cpcloud
added
the
datatypes
cpcloud
force-pushed
the
datafusion-simpler-types-from-query
branch
from
02f1bfc to
fc40cbd
Compare
|
Going to make this only apply to datafusion instead of a broader set of functionality. No reason to do it unless needed/asked for and it's currently only needed in the DataFusion case. |
cpcloud
force-pushed
the
datafusion-simpler-types-from-query
branch
2 times, most recently
from
c82a6b6 to
c7b742a
Compare
cpcloud
force-pushed
the
datafusion-simpler-types-from-query
branch
from
c7b742a to
7c6d312
Compare
jcrist
approved these changes
Jun 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
This PR moves datafusion type system conversion to use PyArrow, which is much
more robust than the SQL hacking we've been doing.
I've added two new methods to support this, required to workaround datafusion inconsistencies in output types:but also potentially useful for ClickHouse
DataType._as_nullable: returns a nullable version of the type, recursively converting for nested types.DataType._as_non_nullable: similar to_as_nullablebut going the other direction, making everything non-nullable.Two analogous methods have been added toSchemaas well.Unit tests are included, including a hypothesis test to cover as many datatypes as possible.
This now is specific to datafusion to avoid an unnecessary maintenance burden for functionality that might never be used.