Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datafusion: SQL projection returns wrong column for partitioned data #1292

Closed
cmackenzie1 opened this issue Apr 16, 2023 · 1 comment · Fixed by #1303
Closed

Datafusion: SQL projection returns wrong column for partitioned data #1292

cmackenzie1 opened this issue Apr 16, 2023 · 1 comment · Fixed by #1303
Labels
binding/rust Issues for the Rust crate bug Something isn't working

Comments

@cmackenzie1
Copy link
Contributor

cmackenzie1 commented Apr 16, 2023 •
edited
Loading

Environment

Delta-rs version: 0.9.0

Binding: rust

Environment:

  • Cloud provider: N/A
  • OS: macOS 13.3.1
  • Other:

Bug

What happened:

Querying a partitioned table using a column projection results in the wrong column being returned.

What you expected to happen:

The correct column projection and data should be returned

How to reproduce it:

See #1293

More details:

Looks like same issue described in #1168
@cmackenzie1 cmackenzie1 added the bug Something isn't working label Apr 16, 2023
cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Apr 16, 2023
@cmackenzie1
issue-1292: sql projection test case
f412c7c 
This commit adds a unit test demonstrating the issue described
in delta-io#1292.
@cmackenzie1 cmackenzie1 mentioned this issue Apr 16, 2023
Closed
cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Apr 16, 2023
@cmackenzie1
issue-1292: sql projection test case
39c9eb3 
This commit adds a unit test demonstrating the issue described
in delta-io#1292.
cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Apr 16, 2023
@cmackenzie1
issue-1292: sql projection test case
2e0640e 
This commit adds a unit test demonstrating the issue described
in delta-io#1292.
@cmackenzie1
Copy link
Contributor Author

I think this might be caused by the values being passed into

delta-rs/rust/src/delta_datafusion.rs

Line 373 in 449fe5f

projection: Option<&Vec<usize>>,

FileScanConfig docs

cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Apr 16, 2023
@cmackenzie1
issue-1292: sql projection test case
8ac6a76 
This commit adds a unit test demonstrating the issue described
in delta-io#1292.
@roeap roeap mentioned this issue Apr 20, 2023
Merged
@rtyler rtyler added the binding/rust Issues for the Rust crate label Apr 25, 2023
wjones127 pushed a commit to roeap/delta-rs that referenced this issue Apr 30, 2023
@cmackenzie1 @wjones127
issue-1292: sql projection test case
3d31c37 
This commit adds a unit test demonstrating the issue described
in delta-io#1292.
rtyler pushed a commit that referenced this issue Apr 30, 2023
@cmackenzie1 @rtyler
issue-1292: sql projection test case
64d6465 
This commit adds a unit test demonstrating the issue described
in #1292.
cmackenzie1 added a commit to cmackenzie1-contrib/delta-rs that referenced this issue Jun 7, 2023
@cmackenzie1
chore: rename test dataset http_requests to match issue numbers
b644d8a 
This results in duplicating the same table for delta-io#1291 and delta-io#1292 but
I think the clarity is worth it.
ChewingGlass pushed a commit to helium/delta-rs that referenced this issue Jun 12, 2023
@cmackenzie1 @ChewingGlass
chore: rename test dataset http_requests to match issue numbers
a7df795 
This results in duplicating the same table for delta-io#1291 and delta-io#1292 but
I think the clarity is worth it.
ChewingGlass pushed a commit to helium/delta-rs that referenced this issue Jun 28, 2023
@cmackenzie1 @ChewingGlass
chore: rename test dataset http_requests to match issue numbers
3612679 
This results in duplicating the same table for delta-io#1291 and delta-io#1292 but
I think the clarity is worth it.
ChewingGlass pushed a commit to helium/delta-rs that referenced this issue Jun 28, 2023
@cmackenzie1 @ChewingGlass
chore: rename test dataset http_requests to match issue numbers
b71309b 
This results in duplicating the same table for delta-io#1291 and delta-io#1292 but
I think the clarity is worth it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
binding/rust Issues for the Rust crate bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: scan partitioned tables with datafusion roeap/delta-rs
2 participants
@rtyler @cmackenzie1