[Spark] Drop Type Widening Table Feature #2720
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spark/src/main/scala/org/apache/spark/sql/delta/PreDowngradeTableFeatureCommand.scala
Show resolved
Hide resolved
|
| This PR includes changes from #2708 which isn't merged yet. |
vkorukanti
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes changes from #2708 which isn't merged yet.
The changes related only to dropping the table feature are in commit e2601a6
Description
This change is part of the type widening table feature.
Type widening feature request: #2622
Type Widening protocol RFC: #2624
It adds the ability to remove the type widening table feature by running the
ALTER TABLE DROP FEATUREcommand.Before dropping the table feature, traces of it are removed from the current version of the table:
REORG TABLEoperation.How was this patch tested?
DeltaTypeWideningSuiteto cover dropping the table feature with tables in various states: with/without files to rewrite or metadata to remove.Does this PR introduce any user-facing changes?
The table feature is available in testing only, there's no user-facing changes as of now.
When the feature is available, this change enables the following user action:
This succeeds immediately if no version of the table contains traces of the table feature (= no type changes were applied in the available history of the table.
Otherwise, if the current version contains traces of the feature, these are removed: files are rewritten if needed and type widening metadata is removed from the table schema. Then, an error
DELTA_FEATURE_DROP_WAIT_FOR_RETENTION_PERIODis thrown, telling the user to retry once the retention period expires.If only previous versions contain traces of the feature, no action is applied on the table, and an error
DELTA_FEATURE_DROP_HISTORICAL_VERSIONS_EXISTis thrown, telling the user to retry once the retention period expires.