misc/metrics: Add auxiliary crate to record events as OpenMetrics

[mxinden]

Motivation

Metrics (e.g. Prometheus or OpenMetrics) enable one to gain a deep understanding of a system, both while developing and operating it.

I find myself re-implementing metric instrumentation for rust-libp2p over and over again. The most advanced metric setup likely lives in Substrate. Other smaller setups can e.g. be found in the Kademlia exporter. Instead of duplicating most of this logic once more for rust-libp2p-relay-server I would like to propose collaborating on a set of base metrics via a new crate here in rust-libp2p - namely libp2p-metrics - providing extensive instrumentation to rust-libp2p users out-of-the-box.

In the long term, I would like to extend this effort beyond the definition and exposition of metrics, all the way to Grafana dashboard definitions and alerting rules. See Substrate/.maintain as an example.

Implementation

As said in #2013 I don't plan to introduce OpenMetrics in each rust-libp2p crate itself, but instead I want to provide an optional auxiliary crate that can live on-the-side and can be used on-demand by recording events emitted by the Swarm and various protocols.

I have not yet found a great abstraction that is both intuitive and extensible. For now I am proposing the below, though I am more than happy for alternative suggestions.

let mut swarm = Swarm::new(
    block_on(libp2p::development_transport(local_key))?,
    Ping::new(PingConfig::new().with_keep_alive(true)),
    local_peer_id,
);

let mut metric_registry = Registry::default();
let metrics = Metrics::new(&mut metric_registry);
thread::spawn(move || block_on(metrics_server(metric_registry)));

block_on(async {
    loop {
        match swarm.next_event().await {
            SwarmEvent::Behaviour(ping_event) => {
                metrics.record(&ping_event);
            }
            swarm_event => {
                metrics.record(&swarm_event);
            }
        }
    }
})

You can find the full example in misc/metrics/examples/metrics.rs in this pull request.

Note, this pull request is using the open-metrics-client crate (authored by me) instead of the more popular prometheus crate. You can find a high-level comparison on tikv/rust-prometheus#392.

Fixes #2013

[mxinden]

The new crate proposed here - libp2p-metrics - is already used by all deployments of https://github.com/mxinden/rust-libp2p-relay-server. To find an abstraction that works for many projects instead of just one, I would like to collaborate with someone, preferably with a large code-base, helping them introduce libp2p-metrics into their project. (It sounds a lot more complicated than it is.)

Any volunteers? @ec2 would this be interesting for you and https://github.com/ChainSafe/forest?

[ec2]

This is very cool! @olibearo from our team will be leading the metrics and instrumentation efforts. I think we would be very down. Let me direct Jorge's attention to this thread so he can chime in.

[dvc94ch]

ipfs-embed uses something prometheus based. while I probably won't change it for the time being, I will eventually rewrite it and make the core abstraction provided to applications a stream instead of a block and remove all the complex garbage collection etc. this crate does seem useful, thanks!

[mxinden]

Great to hear @dvc94ch, let me know in case you every end up using it.