Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow StreamConsumer to reject partition from KafkaIngester #5751

Open
cpwright opened this issue Jul 10, 2024 · 0 comments
Open

Allow StreamConsumer to reject partition from KafkaIngester #5751

cpwright opened this issue Jul 10, 2024 · 0 comments
Assignees
Labels
feature request New feature or request triage

Comments

@cpwright
Copy link
Contributor

As an integrator, I would like more control over the StreamConsumers that are created from the KafkaIngester. In particular, I may find that some partitions cannot be ingested by my backend.

If I throw an Exception, then the entire ingestion process is going to be terminated. If I return a do-nothing ingester, then there will still be additional effort expended processing those partitions.

Allowing the Function<TopicPartition, KafkaRecordConsumer> to return null and behave similarly to filtering out the partition, would provide backwards compatible behavior and more flexibility.

@cpwright cpwright added feature request New feature or request triage labels Jul 10, 2024
@cpwright cpwright self-assigned this Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request triage
Projects
None yet
Development

No branches or pull requests

1 participant