-
Notifications
You must be signed in to change notification settings - Fork 510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metrics for bytes and objects written during compaction #360
Add metrics for bytes and objects written during compaction #360
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we drop metricCompactionDuration
? I feel like that metric loses it's meaning with the variable block compactions.
Also, what do you think about tempodb_compaction_blocks_total{level=''}
. A counter that increments once for every processed block of each level. We could drop the duration histogram while still having some vision on which compactors are compacting blocks at which levels.
Oh, and feel free to add any of this to the operational dashboard that makes sense. I think there's a compaction section now.
Agree, added the level label and new metric, and added both new metrics to the operational dashboard. I am not 100% satisfied with I'm inclined to leave the duration metric for now, while we settle on the best KPIs for compaction. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of final touches and it will be ready to merge. Also make sure to list the removed metric in the changelog. Thanks!
…objects_written metrics when flushing for efficiency
d942072
to
47d7f97
Compare
What this PR does:
Adds three new new metrics to track compaction performance and help estimate capacity. The metrics include a
level
label which is the input compaction level.tempodb_compaction_objects_written
andtempodb_compaction_bytes_written
which are updated when the compactor flushes to the backend.tempodb_compaction_blocks_total
Count of input blocks compacted.Removes tempodb_compaction_duration_seconds metric as it is obsoleted by the new metrics.
Which issue(s) this PR fixes:
n/a
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]