Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changed Fetch.default value from 32KB to 1MB #1024

Merged
merged 1 commit into from
Jan 18, 2018
Merged

Conversation

bshafiee
Copy link
Contributor

In my experiments of consuming very busy topics, I noticed this parameter makes a huge difference. Bumping up the default value from 32KB to 1MB almost increased consumption rate by 5X. 1MB seems to be the default value used in librdkafka as well.

@bshafiee bshafiee requested a review from eapache January 18, 2018 19:15
Copy link
Contributor

@eapache eapache left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To capture a bit more of our offline conversation here:
I originally chose 32KB because I figured that would be the point of peak performance - any response over that size will be fragmented at the network layer anyway and I assumed that would limit the performance gains from further batching.

Clearly this was false - the cost we reduce through larger batches is not just per-packet routing costs but more per-request latency costs (and also the number of disk reads on the broker).

@eapache eapache merged commit 1abfd98 into master Jan 18, 2018
@eapache eapache deleted the behrooz_Fetch_default branch January 18, 2018 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants