-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disallow dropping partition column in Iceberg table explicitly #21294
Disallow dropping partition column in Iceberg table explicitly #21294
Conversation
Isn't it true that this issue only affects prior (historical) specs, and not the latest? So for example, it doesn't affect brand new tables, and it doesn't affect newly added columns? |
That's right, because dropping columns used in current partition spec would explicitly fail in Iceberg implementation. So when we try to drop a partition column, we must firstly drop it from current partition spec. That means, the column exists in a historical spec now. Then we will meet various problems subsequently after dropping it from schema. In my experiment, simultaneously drop the column from partition spec and schema in a single transaction would meet problems in the issue too. Furthermore, this PR can also prevent Iceberg table falling into this situation in case that, for example, if we support dropping column from partition spec only. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
.flatMap(partitionSpec -> partitionSpec.fields().stream()) | ||
.anyMatch(field -> field.sourceId() == handle.getId()); | ||
if (shouldNotDropPartitionColumn) { | ||
throw new PrestoException(NOT_SUPPORTED, "This connector does not support drop partition columns"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Personally, I would make the error message more specific
throw new PrestoException(NOT_SUPPORTED, "This connector does not support drop partition columns"); | |
throw new PrestoException(NOT_SUPPORTED, "This connector does not support dropping columns which exist in any of the table's partition specs"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed!
ab6c619
to
69b37e3
Compare
69b37e3
to
035f136
Compare
Description
When we try to drop a partition column in Iceberg table, we should firstly remove it from the partition spec, then drop the column in table schema. But currently Iceberg have some problems in this scenario, see issue apache/iceberg#4563. This PR explicitly disallow drop partition column until Iceberg fixed it.
Contributor checklist
Release Notes