Add prometheus metric for per partition high watermark #11148
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nharring-adjacent
force-pushed
the
kafkaMetrics
branch
from
c60e4a4 to
275bad5
Compare
nharring-adjacent
force-pushed
the
kafkaMetrics
branch
from
a2973db to
8d28582
Compare
benesch
reviewed
Mar 14, 2022
nharring-adjacent
force-pushed
the
kafkaMetrics
branch
from
9084fc7 to
d737730
Compare
guswynn
reviewed
Mar 14, 2022
benesch
reviewed
Mar 15, 2022
added 2 commits
March 15, 2022 15:16
benesch
reviewed
Mar 16, 2022
| ------------------ | ||
| 0 2 | ||
| 1 2 | ||
| 2 1 |
There was a problem hiding this comment.
This test looks great. Glad it already paid some dividends, too!
added 2 commits
March 16, 2022 10:04
wangandi
pushed a commit
to wangandi/materialize
that referenced
this pull request
Apr 11, 2022
…c#11148) * initial addition of broker high watermark per partition * lint * fix rebase * clean up deserialization * use cursor instead of raw vec * Lint * address feedback * add tests, fix issues found by tests * lint * feedback * lint
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As mentioned in https://github.com/MaterializeInc/cloud/issues/2355 there is a desire to have the broker reported high watermark for each partition available in metrics which was lost when the interaction with kafka was reworked. This PR adds the metric
mz_kafka_partition_offset_maxwith tags fortopic,source_id,source_instanceandpartition_idto the set exported by prometheus which also puts it into themz_metricssystem table.Motivation
https://github.com/MaterializeInc/cloud/issues/2355
Tips for reviewer
No, high watermark is not the correct spelling but its what kafka and rdkafka use so we will all have to wonder exactly what happens to a watermark to get it high or low.
Testing
Release notes
This PR includes the following user-facing behavior changes: