metrics: improve reading Go runtime metrics #25886
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
GC pause metric doesn't work yet |
fjl
force-pushed
the
metrics-runtime-metrics
branch
2 times, most recently
from
24dff68
to
825acd1
Compare
The bytes-based metric is better because it includes 'tiny' objects. It's also easier to understand.
The result of Histogram methods should never be Inf or NaN because the metrics exporters can't handle these values.
holiman
approved these changes
Nov 10, 2022
shekhirin
pushed a commit
to shekhirin/go-ethereum
that referenced
this pull request
Jun 6, 2023
This changes how we read performance metrics from the Go runtime. Instead of using runtime.ReadMemStats, we now rely on the API provided by package runtime/metrics. runtime/metrics provides more accurate information. For example, the new interface has better reporting of memory use. In my testing, the reported value of held memory more accurately reflects the usage reported by the OS. The semantics of metrics system/memory/allocs and system/memory/frees have changed to report amounts in bytes. ReadMemStats only reported the count of allocations in number-of-objects. This is imprecise: 'tiny objects' are not counted because the runtime allocates them in batches; and certain improvements in allocation behavior, such as struct size optimizations, will be less visible when the number of allocs doesn't change. Changing allocation reports to be in bytes makes it appear in graphs that lots more is being allocated. I don't think that's a problem because this metric is primarily interesting for geth developers. The metric system/memory/pauses has been changed to report statistical values from the histogram provided by the runtime. Its name in influxdb has changed from geth.system/memory/pauses.meter to geth.system/memory/pauses.histogram. We also have a new histogram metric, system/cpu/schedlatency, reporting the Go scheduler latency.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes how we read performance metrics from the Go runtime. Instead of
using
runtime.ReadMemStats, we now rely on the API provided by package runtime/metrics.Here's why:
ReadMemStatsis a stop-the-world operation, It requires halting all goroutines,resulting in very unpredictable latency. Under high load, it can sometimes take milliseconds
to run, and all other work will be interrupted while it waits.
runtime/metrics also provides more accurate information. For example, the new interface
has better reporting of memory use. In my testing, the reported value of
heldmemorymore accurately reflects the usage reported by the OS.
The semantics of metrics
system/memory/allocsandsystem/memory/freeshavechanged to report amounts in bytes.
ReadMemStatsonly reported the count of allocationsin number-of-objects. This is imprecise: 'tiny objects' are not counted because the runtime
allocates them in batches; and certain improvements in allocation behavior, such as struct
size optimizations, will be less visible when the number of allocs doesn't change.
Changing allocation reports to be in bytes will make it appear in graphs that lots more is
being allocated. I don't think that's a problem because this metric is primarily interesting
for geth developers.
I have also deleted a lot of old and unused code for reading runtime metrics. The functions
in
metrics/runtime*.gowere never used in go-ethereum.