Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeed: add protobuf as row emission format #68554

Open
sheaffej opened this issue Aug 6, 2021 · 2 comments
Open

changefeed: add protobuf as row emission format #68554

sheaffej opened this issue Aug 6, 2021 · 2 comments
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc

Comments

@sheaffej
Copy link
Collaborator

sheaffej commented Aug 6, 2021

Epic: CRDB-9086

Customer requests to add protobuf row emission format for Changefeeds, for example when emitting to Kafka.

https://www.cockroachlabs.com/docs/stable/create-changefeed.html

The CREATE CHANGEFEED statement creates a new enterprise changefeed, which targets an allowlist of tables, called "watched rows". Every change to a watched row is emitted as a record in a configurable format (JSON or Avro) to a configurable sink (Kafka or a cloud storage sink). You can create, pause, resume, or cancel an enterprise changefeed.

Customer requests consideration of adding Protobuf as a row emission format to Kafka because they use Protobufs for all messages between systems in the their organization. They can work around this and use Avro by writing an Avro-to-Protobuf converter, however, this will be an outlier piece of code that will be awkward to maintain as they have standardized on Protobuf as the data exchange format.

Jira issue: CRDB-9088

@sheaffej sheaffej added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-cdc Change Data Capture labels Aug 6, 2021
@blathers-crl blathers-crl bot added the T-cdc label Aug 6, 2021
@amruss
Copy link
Contributor

amruss commented Aug 9, 2021

Putting on the backlog for now and will revisit if there is a strong customer need - Would be tricky due to schema registry edginess.

@shermanCRL shermanCRL added E-starter Might be suitable for a starter project for new employees or team members. E-quick-win Likely to be a quick win for someone experienced. labels Jun 1, 2022
Copy link

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc
Projects
None yet
Development

No branches or pull requests

3 participants