Add Row ID specification to protocol #1610
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
johanl-db
commented
Mar 24, 2023
PROTOCOL.md
Outdated
| or with any other `add` action in the same commit. | ||
| An `add` action contains default generated Row IDs ranging from `add.baseRowId` (inclusive) to `add.baseRowId + add.stats.numRecords` (exclusive) for non-empty files. Empty files contain no default generated Row IDs. | ||
| - `add` actions for new physical files must have a value in the `baseRowId` field that is larger than the `highWaterMark` value in the last committed `rowIdHighWaterMark` action. | ||
| If the table doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1. |
There was a problem hiding this comment.
Suggested change
| If the table doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1. | |
| If the table version doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1. |
|
@johanl-db the design doc is not public available |
Sorry about that, you should now be able to access the document. |
|
@johanl-db is the plan to send the implementation PR soon? Without reference implementation other libraries are more likely to deviate from spec and create compatibility issues. |
Yes, I'm actively working on it and I plan to open the first PRs implementing writing default row ids in the next few days. Support for reading and writing materialized row ids is also in progress though it requires changes in Spark. I started shipping these (e.g. apache/spark#40545) but I'll need to wait for Delta to pick up the changes in a later Spark release before moving forward which may take some time. When that happens, I'll open PRs to support reading and writing materialized row ids. |
felipepessoto
approved these changes
Apr 17, 2023
allisonport-db
pushed a commit
that referenced
this pull request
Apr 24, 2023
This change adds knobs to enable Row IDs when creating a new Delta table, as defined in the Row ID specification (#1610): - Write table feature `rowIds`: Require writers to support row IDs. Used to enable row IDs on newly created tables. - Table property: `rowIds.enabled`: Indicate whether all rows have an assigned row ID. - SQL conf `rowIds.allowForDevOnly`: restrict the use of Row IDs to testing for now. Adding test suite RowIdSuite: - Test enabling Row IDs on a new table succeeds - Test enabling Row IDs on an existing table fails. Closes #1702 GitOrigin-RevId: 961ff72f1ae7abf1f08d53052062ce20669d4aad
allisonport-db
pushed a commit
that referenced
this pull request
Jul 21, 2023
## Description This change is part of implementing row tracking as specified in #1610 and 7272b04. It covers assigning a column name for the materialized Row ID and Row commit version columns by setting them in the table metadata when creating or cloning a table. - Add test suite `rowtracking.MaterializedColumnSuite` to cover assigning materialized column names in various table creation and clone scenarios. Closes #1896 GitOrigin-RevId: 3b963ed7e08524f24d60744160490e41a0aab3e8
This was referenced Apr 8, 2024
Closed
Closed
scottsand-db
pushed a commit
that referenced
this pull request
Apr 9, 2024
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Delta Protocol for Row IDs was introduced in this PR: #1610 Support for writing fresh row IDs / row commit versions was introduced in the following PRs: - #1723 - #1781 - #1896 **This is sufficient to enable row tracking on a table and write to a table that has row tracking enabled** but not to actually read row IDs / row commit versions back, which is also being added in Delta at the moment ([read BaseRowId](283ac02). [read defaultRowCommitVersion](#2795), [read RowId](#2856)...) Using row tracking is currently only allowed in testing, this change allows enabling row tracking outside of testing so that the upcoming Delta 3.2 release includes support for writing to tables with row tracking enabled, making Delta writers future-proof. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Tests have already been added in previous changes, this only flips the switch to let users enabled Row Tracking outside of tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? Users are now able to enable Row Tracking when creating a delta table: ``` CREATE TABLE tbl(a int) USING DELTA TBLPROPERTIES ('delta.enableRowTracking' = 'true') ``` <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
andreaschat-db
pushed a commit
to andreaschat-db/delta
that referenced
this pull request
Apr 16, 2024
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Delta Protocol for Row IDs was introduced in this PR: delta-io#1610 Support for writing fresh row IDs / row commit versions was introduced in the following PRs: - delta-io#1723 - delta-io#1781 - delta-io#1896 **This is sufficient to enable row tracking on a table and write to a table that has row tracking enabled** but not to actually read row IDs / row commit versions back, which is also being added in Delta at the moment ([read BaseRowId](delta-io@283ac02). [read defaultRowCommitVersion](delta-io#2795), [read RowId](delta-io#2856)...) Using row tracking is currently only allowed in testing, this change allows enabling row tracking outside of testing so that the upcoming Delta 3.2 release includes support for writing to tables with row tracking enabled, making Delta writers future-proof. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Tests have already been added in previous changes, this only flips the switch to let users enabled Row Tracking outside of tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? Users are now able to enable Row Tracking when creating a delta table: ``` CREATE TABLE tbl(a int) USING DELTA TBLPROPERTIES ('delta.enableRowTracking' = 'true') ``` <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change updates the Delta protocol to add the specification for the Row ID feature.
Row IDs are integers that are used to uniquely identify rows within a table.
See Row ID design document for additional context: