Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(inputs.kafka_consumer): Option to set default fetch message bytes #11220

Merged
merged 8 commits into from
Jul 25, 2022
Merged

feat(inputs.kafka_consumer): Option to set default fetch message bytes #11220

merged 8 commits into from
Jul 25, 2022

Conversation

AlbertasB
Copy link
Contributor

Required for all PRs:

This allows setting number of message bytes to fetch from the broker in each request (default 1MB). Similar to the JVM's fetch.message.max.bytes. Might be useful in some cases. I needed it specifically to make telegraf consume messages faster from Kafka cluster.

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label May 30, 2022
@sspaink sspaink changed the title feat(inputs/kafka_consumer): allow to set default fetch message bytes feat(inputs.kafka_consumer): Allow to set default fetch message bytes Jun 22, 2022
@sspaink sspaink changed the title feat(inputs.kafka_consumer): Allow to set default fetch message bytes feat(inputs.kafka_consumer): Option to set default fetch message bytes Jun 22, 2022
@sspaink sspaink added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Jun 22, 2022
srebhan
srebhan reviewed Jun 29, 2022
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @AlbertasB, thanks for the PR! I have two small comments. Can you please take a look?

plugins/inputs/kafka_consumer/README.md Outdated
Comment on lines 109 to 111
## negotiating sizes and not actually consuming. Similar to the JVM's
## `fetch.message.max.bytes`.
# fetch_message_bytes = 1048576
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please rename to consumer_fetch_default as this is what you ultimately use it as such.

plugins/inputs/kafka_consumer/kafka_consumer.go Outdated
@@ -43,6 +43,7 @@ type KafkaConsumer struct {
BalanceStrategy string `toml:"balance_strategy"`
Topics []string `toml:"topics"`
TopicTag string `toml:"topic_tag"`
FetchMessageBytes int32 `toml:"fetch_message_bytes"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making this a config.Size so you can specify human-readable sizes like 1MB or 1MiB?

@srebhan
Copy link
Member

srebhan commented Jul 12, 2022

@AlbertasB did you find some time to address my comments? Other than the renaming and type changes this is good and I would really like to see this being merged!

@powersj
Copy link
Contributor

powersj commented Jul 20, 2022

Hi @AlbertasB, it looks like a couple of final changes are required, and then we can do a final review. Can you loop back on this? Thanks!

@powersj powersj added the waiting for response waiting for response from contributor label Jul 20, 2022
@AlbertasB
Copy link
Contributor Author

Hey, thanks for the review and the comments. Somehow I did not get the message :(. I will try to make those changes ASAP. Thanks again :)

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Jul 25, 2022
reimda
reimda suggested changes Jul 25, 2022
View reviewed changes
Copy link
Contributor

@reimda reimda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's looking great, almost ready to merge. It looks like there's an extra file in the PR that needs to be removed: oryxBuildBinary. Maybe a test file that sneaked in?

@AlbertasB
Copy link
Contributor Author

Ooops no idea how that got there, nice catch! Removed :)

@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_i386.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz
static_linux_amd64.tar.gz

@reimda reimda merged commit b1546fe into influxdata:master Jul 25, 2022
@reimda reimda mentioned this pull request Jul 26, 2022
Merged
@sjwang90 sjwang90 mentioned this pull request Sep 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srebhan srebhan srebhan left review comments

@reimda reimda reimda approved these changes

@sspaink sspaink sspaink approved these changes

Assignees
No one assigned
Labels
feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@AlbertasB @srebhan @powersj @sspaink @reimda