[WIP] changefeedccl: support tables with more than one column family #30414
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If a table has multiple column families and only one of them is changed in a given statement, only the kv for that column family will be changed by the sql system. This means we wouldn't have enough data to reconstruct the row. We can fetch the row's entire kv data with a prefix scan that's forced to be at the timestamp of the changed kv. This has two issues - it introduces a blocking kv read (in the common case, this is not as bad as it sounds since this code is likely running on the leaseholder and the data can be served locally) - if N column families were updated at once, then we'll run this process N times and get N exact duplicates in the output Release note (enterprise change): CHANGEFEEDs now support tables with multiple column families, though with degraded performance. Closes cockroachdb#28667
|
This change is |
|
@mrtracy This is slated for 2.2, so there's no rush to get it in. I've been dragging my feet on this because of the followup fetch, but realized it's not so bad since it's likely to be served by the local node. However, I think the duplicates probably make this unmergeable as-is. Thoughts? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a table has multiple column families and only one of them is changed
in a given statement, only the kv for that column family will be changed
by the sql system. This means we wouldn't have enough data to
reconstruct the row.
We can fetch the row's entire kv data with a prefix scan that's forced
to be at the timestamp of the changed kv. This has two issues
bad as it sounds since this code is likely running on the leaseholder
and the data can be served locally)
N times and get N exact duplicates in the output
Release note (enterprise change): CHANGEFEEDs now support tables with
multiple column families, though with degraded performance.
Closes #28667