fix: canonicalization of chunked ExtensionArray #499
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a10y
commented
Jul 22, 2024
robert3005
reviewed
Jul 22, 2024
robert3005
reviewed
Jul 22, 2024
a10y
force-pushed
the
aduffy/compress-ext-chunks
branch
from
991a48e
to
ef6e6d9
Compare
a10y
changed the title
a10y
commented
Jul 23, 2024
lwwmanning
reviewed
Jul 23, 2024
gatesn
reviewed
Jul 23, 2024
| @@ -28,6 +28,10 @@ pub(crate) fn try_canonicalize_chunks( | |||
| chunks: Vec<Array>, | |||
| dtype: &DType, | |||
| ) -> VortexResult<Canonical> { | |||
| if chunks.is_empty() { | |||
There was a problem hiding this comment.
You should add a FLUP to create Canonical::empty(dtype)
There was a problem hiding this comment.
If we end up redesigning extension arrays to contain their storage_dtype as part of the type definition, i'm not sure we'd need it
|
Per discussion on Slack yesterday, I updated the branch to have canonicalization behavior Chunked(Extension(storage)) -> Extension(Chunked(storage)). As part of that, I added call to .into_canonical() for the DateTimePartsCompressor. |
robert3005
approved these changes
Jul 24, 2024
|
Should change the commit message since the behaviour is different than what we started with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since #346, we've been canonicalizing chunked ExtensionArray's incorrectly. Rather than unwrapping each chunk to its backing storage array we've just been building a new chunkedarray of the ExtensionArray chunks.
Before #480 , this was actually causing DateTimePartsCompressor to fail in can_compress, so we weren't going down the compression codepath.
The Fix
Fix
into_canonicalso that when we encounter aChunkedArray(ExtensionArray(storage))to yield anExtensionArray(ChunkedArray(storage))where each chunk is canonicalized first (e.g. if you have chunks of DateTimePartsArray you will end up with chunks of ExtensionArray(i64)).Fix the DateTimePartsCompressor to canonicalize its input, to handle that case where it may be a ChunkedArray.