Support StringViewArray interop with python: fix lingering C Data Interface issues for *ViewArray #6368
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Keeping this in draft until I can do some manual verification that it works |
9 tasks
This was referenced Sep 9, 2024
alamb
reviewed
Sep 11, 2024
There was a problem hiding this comment.
Thank you @a10y -- this code looks great to me. I am not a low level expert in this area but I read it closely and it looks good. I found the code clear and easy to follow and the comments really helped. 🏆
I think the PR needs a few more tests (I commented on which below) but otherwise is ready to go.
4 tasks
alamb
changed the title
This was referenced Sep 11, 2024
|
@alamb I believe I've implemented all of the requested tests, thanks for reviewing! |
alamb
approved these changes
Sep 11, 2024
| assert_eq!(mixed_one_variadic.data_buffers().len(), 1); | ||
| run_test_case!(mixed_one_variadic); | ||
|
|
||
| // inlined + non-inlined, 2 variadic buffers. |
| } | ||
|
|
||
| fn binary_view_column(num_variadic_buffers: usize) -> BinaryViewArray { | ||
| let long_scalar = b"but soft what light through yonder window breaks".as_slice(); |
|
Thanks again @a10y |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6366
Rationale for this change
Previously, StringViewArrays and BinaryViewArrays sent to PyArrow will panic.
If you use the updated
test_to_pyarrow()from this PR without the additional changes, it will fail.What changes are included in this PR?
The specification states that the C Data Interface format for StringViewArray and BinaryViewArray includes an extra
variadic_buffer_sizesbuffer at the end of the various output values.We were not including that before. There was a prior PR to add it for the IPC format, but not for the C Data Interface.
I cribbed the relevant C++ code for this: https://github.com/apache/arrow/blob/ab0a40ee34217070f14027776682074c55d0b507/cpp/src/arrow/c/bridge.cc#L584-L609
Are there any user-facing changes?
It should now be possible to share StringViewArray and BinaryViewArray over C Data Interface.