Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error reading from delta table with >= 100 commits #308

Open
zachschuermann opened this issue Aug 8, 2024 · 2 comments
Open

Error reading from delta table with >= 100 commits #308

zachschuermann opened this issue Aug 8, 2024 · 2 comments
Assignees

Comments

@zachschuermann
Copy link
Collaborator

zachschuermann commented Aug 8, 2024

From duckdb/duckdb#12983

InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: './test_table/'): Hit error: 2 (ArrowError) with message (Invalid argument error: Incorrect datatype for StructArray field "partitionValues", expected Map(Field { name: "entries", data_type: Struct([Field { name: "keys", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "values", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, false) got Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false))
@zachschuermann zachschuermann self-assigned this Aug 8, 2024
@ion-elgreco
Copy link
Contributor

@zachschuermann it's a problem when reading the checkpoint, the engine assumes Map type is constructed with Entries, keys, values but it's receiving entries, key value

@zachschuermann
Copy link
Collaborator Author

Yep looks like it. I'm guessing we aren't reading the checkpoint when @samansmink and I are failing to repro the issue. First thing I'd like to do is understand the repro then fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants