Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changefeedccl: add unordered option to gc pubsub sink #80884

Closed
amruss opened this issue May 2, 2022 · 3 comments
Closed

changefeedccl: add unordered option to gc pubsub sink #80884

amruss opened this issue May 2, 2022 · 3 comments
Assignees
@HonoreDB
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) docs-done docs-known-limitation E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc

Comments

@amruss
Copy link
Contributor

amruss commented May 2, 2022 •
edited by exalate-issue-sync bot
Loading

In 22.1, we introduced the gc pubsub sink in beta. Currently, you MUST specify a region to send changefeed data to, this is to keep our ordering guarantees. We should add an unordered option, which also makes specifying a region optional.

We should add the configuration:

  • ordered=false
  • default value: true
  • type: query parameter
  • applies to: gc pubsub only

Jira issue: CRDB-15467

Epic CRDB-13237
@amruss amruss added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-cdc Change Data Capture T-cdc labels May 2, 2022
@blathers-crl
Copy link

blathers-crl bot commented May 2, 2022

cc @cockroachdb/cdc

@amruss amruss added E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. labels May 4, 2022
@jlinder jlinder added sync-me and removed sync-me labels May 20, 2022
@lopezator
Copy link
Contributor

This would be great for us, suscribing to the issue.

@HonoreDB HonoreDB self-assigned this Feb 5, 2023
@HonoreDB HonoreDB mentioned this issue Feb 5, 2023
Merged
HonoreDB added a commit to HonoreDB/cockroach that referenced this issue Feb 10, 2023
@HonoreDB
changefeedccl: add unordered flag
a79eaf3 
This PR adds the WITH unordered changefeed option,
which relaxes our constraints on configuration meant
to preserve end-to-end ordering guarantees. Followup
PRs will use this in different ways, but this one just
removes the requirement to specify a region in gcpubsub.

Addresses cockroachdb#80884.
Informs cockroachdb#54461.

Release note (enterprise change): Changefeeds with the WITH unordered flag may use multiregion Google Cloud pubsub topics.
craig bot pushed a commit that referenced this issue Feb 13, 2023
@HonoreDB
Merge #96567
cdaf027 
96567: changefeedccl: add unordered flag r=[miretskiy] a=HonoreDB

This PR adds the WITH unordered changefeed option, which relaxes our constraints on configuration meant to preserve end-to-end ordering guarantees. Followup PRs will use this in different ways, but this one just removes the requirement to specify a region in gcpubsub.

Release note (enterprise change): Changefeeds with the WITH unordered flag may use multiregion Google Cloud pubsub topics.

Addresses #80884. This is also motivated by #54461.

Co-authored-by: Aaron Zinger <zinger@cockroachlabs.com>
@miretskiy
Copy link
Contributor

Already supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@HonoreDB HonoreDB

Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) docs-done docs-known-limitation E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc
Projects
No open projects
[DEPRECATED] CDC
Status: Closed
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jlinder @HonoreDB @lopezator @amruss @miretskiy