feat(memtable): add pyarrow.dataset support #10206
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
tests
|
I'm going to xfail polars for now -- I think we can strip out all of the custom in-memory reading stuff and that will make this work, but that seems like a separate PR |
jcrist
reviewed
Sep 24, 2024
jcrist
reviewed
Sep 24, 2024
gforsyth
force-pushed
the
dataset_memtable
branch
from
6a0dc25
to
ac36d7a
Compare
1 task
1 task
cpcloud
approved these changes
Sep 26, 2024
This adds support for using `memtable` with `pyarrow.dataset.Dataset` objects. Mostly this looks the same as the `PyArrowTableProxy`. Some deviations: - An additional `to_pyarrow_lazy` method for grabbing the underlying object without materializing it, for backends that can perform pushdowns (currently only enabled with DuckDB). - Datasets are hashable objects, so we override the PseudoHashable `hash` implementation (this seemed like less trouble than special-casing this one memtable everywhere else we look for `TableProxy` objects) - I've added a `cached_property` so we can avoid repeatedly materializing the data -- this does _not_ apply to the `lazy` view.
cpcloud
force-pushed
the
dataset_memtable
branch
from
2f69cde
to
78a9462
Compare
cpcloud
force-pushed
the
dataset_memtable
branch
from
8af5df7
to
b938581
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
This adds support for using
memtablewithpyarrow.dataset.Datasetobjects. Mostly this looks the same as the
PyArrowTableProxy.Some deviations:
to_pyarrow_lazymethod for grabbing the underlyingobject without materializing it, for backends that can perform
pushdowns (currently only enabled with DuckDB).
hashimplementation (this seemed like less trouble than special-casingthis one memtable everywhere else we look for
TableProxyobjects)cached_propertyso we can avoid repeatedlymaterializing the data -- this does not apply to the
lazyview.Issues closed
Resolves #10082