changefeedccl: support tables with more than one column family #28667
Assignees
Labels
A-cdc
Change Data Capture
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Comments
danhhz
added
C-enhancement
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Sep 19, 2018
If a table has multiple column families and only one of them is changed in a given statement, only the kv for that column family will be changed by the sql system. This means we wouldn't have enough data to reconstruct the row. We can fetch the row's entire kv data with a prefix scan that's forced to be at the timestamp of the changed kv. This has two issues - it introduces a blocking kv read (in the common case, this is not as bad as it sounds since this code is likely running on the leaseholder and the data can be served locally) - if N column families were updated at once, then we'll run this process N times and get N exact duplicates in the output Release note (enterprise change): CHANGEFEEDs now support tables with multiple column families, though with degraded performance. Closes cockroachdb#28667
|
This is quite unfortunately not going to make 2.2 |
|
It would also be nice to get an envelope option that includes (only) what the affected column family contains. This should be relatively easy (mostly gating multi-family tables behind the two restrictive envelopes, and formatting for the envelope). |
craig bot
pushed a commit
that referenced
this issue
Feb 22, 2022
73995: docs: jobs explainer doc r=samiskin a=samiskin Similar to "Life of a Query", this change adds an in-depth rundown of how the job system functions and the various pieces in the code that drive it and use it. Release note: None 75430: cdc: refactor to replace map of changefeed targets with array r=[miretskiy] a=HonoreDB Necessary intermediate step towards changefeeds on things other than full tables. Previously, changefeed targets were serialized as a map of TableID->StatementTimeName. This keeps that for now for compatibility but adds in a list of target specifications that's more extendable, and changes the API for everything that will eventually need to care (for example, an encoder will need to know that it's encoding a [column family](#28667)). Still needs a test with the old-style payload. Release note: None 75965: ui: Add column selector and filter to sessions page r=gtr a=gtr Closes [#73463](https://cockroachlabs.atlassian.net/browse/CRDB-11600) Previously, the sessions page was missing the column selector and filter components that the stmts and txns page have. This change adds both components in addition to new columns in the sessions overview page and edits to the sessions details page. Release note (ui change): added column selector, filters, new columns to the sessions page and sessions details. Screenshots: Sessions overview page:  Sessions overview page with column selector:  Sessions overview page with results from column selector:  Sessions overview page with filter:  Sessions overview page with results from filter:  Sessions detail page:  Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com> Co-authored-by: Aaron Zinger <zinger@cockroachlabs.com> Co-authored-by: Gerardo Torres <gerardo.torrescastro@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Changefeeds currently work only for tables with exactly one column family, which is the default for all tables created since sometime before 1.0. But some users may have used this feature and we'll need to do some work before changefeeds work on their tables. This will require a followup fetch for the data in the other column families, so even once it's implemented, the changefeed will be somewhat slower.
One possible workaround in the meantime is to allow for
envelope=key_onlychangefeeds on these tables, which is easy to implement and doesn't need the followup fetch.gz#7511
Epic CRDB-1660
gz#5472
Jira issue: CRDB-4901
The text was updated successfully, but these errors were encountered: