Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature benchmark: add scenario with many Kafka sources in the same cluster #27711

Conversation

nrainer-materialize
Copy link
Contributor

@nrainer-materialize nrainer-materialize commented Jun 17, 2024

@nrainer-materialize nrainer-materialize self-assigned this Jun 17, 2024
@nrainer-materialize nrainer-materialize added the T-testing Theme: tests or test infrastructure label Jun 17, 2024
@nrainer-materialize nrainer-materialize force-pushed the feature-benchmark/many-kafka-sources branch 2 times, most recently from 601f107 to 6868c4d Compare June 18, 2024 06:08
@nrainer-materialize nrainer-materialize force-pushed the feature-benchmark/many-kafka-sources branch from 6868c4d to bcedba0 Compare June 18, 2024 08:11
@nrainer-materialize nrainer-materialize marked this pull request as ready for review June 18, 2024 08:12
@nrainer-materialize
Copy link
Contributor Author

nrainer-materialize commented Jun 18, 2024

I am not 100% sure if this tests what we intend. I am a bit surprised by the low wallclock duration, which is < 10 seconds...
It is either extremely fast or some relevant logic happens already outside of /* A */ [...] /* B */.

@nrainer-materialize nrainer-materialize force-pushed the feature-benchmark/many-kafka-sources branch from bcedba0 to a32e70b Compare June 24, 2024 10:48
Copy link
Contributor

@def- def- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try increasing the scale? If with a larger scale it gets slower I'm ok with this.

@nrainer-materialize nrainer-materialize force-pushed the feature-benchmark/many-kafka-sources branch from a32e70b to ecc93bb Compare June 25, 2024 09:28
@nrainer-materialize
Copy link
Contributor Author

There is a difference when using more sources.

With scale 2.5 (316 sources):

ManyKafkaSourcesOnSameCluster       | wallclock       |          25.569 |          24.310 |      no       | 5.2 pct   more/slower
ManyKafkaSourcesOnSameCluster       | messages        |            None |            None |      no       | N/A
ManyKafkaSourcesOnSameCluster       | memory_mz       |        4684.448 |        4555.702 |      no       | 2.8 pct   more/slower
ManyKafkaSourcesOnSameCluster       | memory_clusterd |          64.383 |          61.665 |      no       | 4.4 pct   more/slower

With scale 2.0 (100 sources):

ManyKafkaSourcesOnSameCluster       | wallclock       |           9.057 |           8.760 |      no       | 3.4 pct   more/slower
ManyKafkaSourcesOnSameCluster       | messages        |            None |            None |      no       | N/A
ManyKafkaSourcesOnSameCluster       | memory_mz       |        2271.652 |        2024.651 |    !!YES!!    | 12.2 pct   more/slower
ManyKafkaSourcesOnSameCluster       | memory_clusterd |          63.248 |          63.381 |      no       | 0.2 pct   less/faster

@nrainer-materialize nrainer-materialize merged commit 79bf684 into MaterializeInc:main Jun 25, 2024
18 checks passed
@nrainer-materialize nrainer-materialize deleted the feature-benchmark/many-kafka-sources branch June 25, 2024 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-testing Theme: tests or test infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants