Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix KafkaInputFormat when used with Sampler API #13900

Merged
merged 3 commits into from
Mar 9, 2023

Conversation

clintropolis
Copy link
Member

@clintropolis clintropolis commented Mar 8, 2023

Fixes #13879.

Description

Fixes issues when using KafkaInputFormat with the sampler API. My initial assessment in #13879 was only partially correct. The implementation of the sample method in KafkaInputReader also wasn't really correct since by just using the normal read path it did not correctly capture ParseException and so would fail immediately. I've adjusted the sample method to delegate to the underlying readers sample, which will correctly handle these, and changed the blending to occur on the InputRowListPlusRawValues of the underlying sample call instead.

Release note

The KafkaInputFormat now supports using the sampler API /druid/indexer/v1/sampler.


This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@gianm gianm merged commit c7f4bb5 into apache:master Mar 9, 2023
@clintropolis clintropolis deleted the kafka-input-format-sampler-fix branch March 9, 2023 00:55
317brian pushed a commit to 317brian/druid that referenced this pull request Mar 10, 2023
* fix KafkaInputFormat when used with Sampler API

* handle key format sampling the same as value format sampling
@clintropolis clintropolis added this to the 26.0 milestone Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sampler API ClassCastException when using KafkaInputFormat
2 participants