CDC: Add GitHub Tracking Issues to Limitations

[kathancox]

Kathryn Hancox (kathancox) commented:

Currently the CDC limitations do not have any GitHub Tracking Issues as per the style for our Known Limitations.

This issue is to add Tracking issues to them where possible. Where there isn't a tracking issue and we're not planning to work on the product in this area, I propose moving to a Considerations section... similar to here on the IMPORT INTO page.

Jira Issue: DOC-1112

[exalate-issue-sync]

Kathryn Hancox (kathancox) commented:
Current changefeed limitations that do not have tracking issues are as follows. Making notes of possible tracking issues that need to be verified. Any without either 1) I couldn’t find a tracking issue, or 2) Might not be limitations — that is, we’re maybe not intending to resolve the “limitation” and it’s actually more of a guideline/consideration for another section:

• Changefeeds only work on tables with a single column family (which is the default for new tables). changefeedccl: support tables with more than one column family cockroach#28667 {color:#6554c0}this is a pretty old issue, also wondering if the 22.1 multi-column work will affect this limitation.{color}
• Changefeeds do not share internal buffers, so each running changefeed will increase total memory usage. To watch multiple tables, we recommend creating a changefeed with a comma-separated list of tables. (2)?
• Many DDL queries (including {{TRUNCATE}} and {{DROP TABLE}}) will cause errors on a changefeed watching the affected tables. You will need to start a new changefeed.
• Changefeeds cannot be backed up or restored. (2)?
• Partial or intermittent sink unavailability may impact changefeed stability; however, ordering guarantees will still hold for as long as a changefeed remains active.
• Changefeeds cannot be altered. To alter, cancel the changefeed and create a new one with updated settings from where it left off. changefeedccl: implement the various ALTER CHANGEFEED described in the RFC cockroach#28668 {color:#6554c0}Again a pretty old issue, so wondering if this is still relevant?{color}
• Additional target options will be added, including partitions. {color:#6554c0}I don’t feel like this is a limitation, necessarily? {color}
• When an {{IMPORT INTO}} statement is run, changefeed jobs targeting that table will fail. {color:#6554c0}(2){color} {color:#6554c0}I think this might be just a guideline? {color}
• Using a cloud storage sink only works with {{JSON}} and emits newline-delimited json files. Support Apache Parquet format for cloud storage sinks cockroach#59819 {color:#6554c0}Maybe this issue, since it is a new format, is there an overarching issue for new emitted formats?{color}
• Webhook sinks only support HTTPS. Use the {{insecure_tls_skip_verify}} parameter when testing to disable certificate verification; however, this still requires HTTPS and certificates.
• Currently, webhook sinks only have support for emitting {{JSON}}.
• There is no concurrency configurability for webhook sinks.
• {{ site.data.products.enterprise }} changefeeds are currently disabled for {{ site.data.products.serverless }} clusters. Core changefeeds are enabled.

[exalate-issue-sync]

Abbey Russell (amruss) commented:
Changefeeds only work on tables with a single column family: limitation, it seems like we already have an issue for this one

Changefeeds do not share internal buffers, so each running changefeed will increase total memory usage: this is a consideration, not something that will change

Many DDL queries (including {{TRUNCATE}} and {{DROP TABLE}}) will cause errors on a changefeed watching the affected tables: I would put this is consideration for now. I don’t think it’s something we can fix and we certainly have no plans to try.

Changefeeds cannot be backed up or restored: This is probably a limitation, we could and probably eventually will do something about this - needs github issue

Partial or intermittent sink unavailability may impact changefeed stability; however, ordering guarantees will still hold for as long as a changefeed remains active: This is a consideration you should plan/configure against.

Changefeeds cannot be altered: limitation, something we are planning to address. I think the issue is fine.

Additional target options will be added, including partitions: I think this is a limitation if we rephrase to “target options limited to tables currently”. Needs issue.

When an {{IMPORT INTO}} statement is run, changefeed jobs targeting that table will fail: I’d put this in considerations since I don’t think we’re going to fix this.

Using a cloud storage sink only works with {{JSON}} and emits newline-delimited json files: Definitely limitation. Hmm I don’t know about that issue. I think it implies that Apache Parque is the first format we’ll support after JSON, which I don’t want to commit to.

Webhook sinks only support HTTPS. Use the {{insecure_tls_skip_verify}} parameter when testing to disable certificate verification; however, this still requires HTTPS and certificates: Hmm definitely a limitation. But we also made a deliberate choice to only support https right now. Probably should just file and issue and backlog it.

Currently, webhook sinks only have support for emitting {{JSON}}: Limitation, though maybe should just have the same github issue as cloud storage.

There is no concurrency configurability for webhook sinks: Limitation, I think we already have an issue for this I’ll look.

Ent changefeeds are currently disabled for serverless clusters. Core changefeeds are enabled: limitation we’re working on right now 🙂. I’ll find an appropriate issue

[exalate-issue-sync]

Abbey Russell (amruss) commented:
Changefeeds cannot be backed up or restored: cockroachdb/cockroach#73434

Changefeeds cannot be altered: cockroachdb/cockroach#28668

Additional target options will be added, including partitions: cockroachdb/cockroach#73435

Using a cloud storage sink only works with {{JSON}} and emits newline-delimited json files: cockroachdb/cockroach#73432

Webhook sinks only support HTTPS: cockroachdb/cockroach#73431

Currently, webhook sinks only have support for emitting {{JSON: https://github.com/cockroachdb/cockroach/issues/73432}}: cockroachdb/cockroach#73432

There is no concurrency configurability for webhook sinks: cockroachdb/cockroach#73430

Ent changefeeds are currently disabled for serverless clusters. Core changefeeds are enabled: cockroachdb/cockroach#73429