Performance Profiling

[SurfingNerd]

We need profiling data of a running process as part of the validator set.

[dforsten]

If using flamegraph, avoid resolving inline functions, it calls the addr2line process for every sample, which takes hours on large code bases like OpenEthereum.

This is a performance issue in "perf" itself, and is fixed in version 5.16. Ubuntu up to version 22.04 installs 5.15 still, meaning this type of analysis is not available to us at this point.

Alternatives would be of course to get 5.16 compiled and working with Ubuntu 22.04, but that should be attempted only in case we absolutely need this type of analysis.

The issue is tracked on GitHub:
flamegraph-rs/flamegraph#74

[dforsten]

The easiest way to get performance analysis is to use the "flamegraph" cargo plugin as described in the Rust Performance Book:

https://nnethercote.github.io/perf-book/profiling.html

Use it with the -no-inline option until we have a "perf" version >= 5.16 available.

[dforsten]

There is a “flamegraph” plugin for cargo, install it with:

cargo install flamegraph

To attach it to a running process (using “--root” option for elevated sampling rights):

flamegraph --root -o hbbft_flamegraph.svg --no-inline --pid 54948

Note:
You can use the “Hotspot” tool to load the “perf.data” file to analyze the gathered sample data. Sometimes it is easier to read the data using the call hierarchy compared to the SVG file generated by the flamegraph tool.

[dforsten]

Do not forget to use:

export RUSTFLAGS="-C target-cpu=native"

It greatly improves performance on the tested machines so far.

[SurfingNerd]

great, thanks