Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix writing incorrect results when partition order is different in Delta Lake #13509

Merged
merged 2 commits into from
Aug 8, 2022

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Aug 5, 2022

Description

CREATE TABLE test_delta (
   id integer,
   first varchar,
   second varchar
)
WITH (
   partitioned_by = ARRAY['second', 'first']
);

This PR fixes wrong partitionValues field and changes directory structure (first=x/second=ysecond=y/first=x). The new directory structure is same as Spark.

Fixes #13505

Documentation

(x) No documentation is needed.

Release notes

(x) Release notes entries required with the following suggested text:

# Delta Lake
* Fix writing incorrect results when the partition column order is different from the table definition. ({issue}`13505`)

@cla-bot cla-bot bot added the cla-signed label Aug 5, 2022
@ebyhr ebyhr force-pushed the ebi/delta-partition-incorrect branch from 155d40d to 32f2814 Compare August 5, 2022 09:32
@ebyhr ebyhr force-pushed the ebi/delta-partition-incorrect branch from 2af8ef7 to 5d6e5b3 Compare August 6, 2022 03:03
@findepi
Copy link
Member

findepi commented Aug 6, 2022

is the CI failure related?

@ebyhr
Copy link
Member Author

ebyhr commented Aug 6, 2022

@findepi No, it's "Table being modified concurrently" error in existing tests.

@ebyhr ebyhr requested a review from findepi August 8, 2022 05:10
@ebyhr ebyhr merged commit ea15ea8 into trinodb:master Aug 8, 2022
@ebyhr ebyhr deleted the ebi/delta-partition-incorrect branch August 8, 2022 08:16
@ebyhr ebyhr mentioned this pull request Aug 8, 2022
@github-actions github-actions bot added this to the 393 milestone Aug 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Delta Lake broken when partition column order does not match table column order
3 participants