Collecting Process Metrics

[jbenet]

We're getting to the point where we want to collect a large swath of metrics. This is for use cases like:

• showing stats on user dashboards (cli + webui)
• making graphs
• taking actions on reaching certain thresholds
• bitswap-ml

So far, some metrics work includes #457 and #1163. Interested people include: @whyrusleeping @lgierth @rht @Heems @jbenet

We should use this issue to track:

• problems we want to solve
• potential solutions
• other issues to work on.

[jbenet]

As goals, i think we should try for something:

• uninvasive (dont change all the interfaces everywhere)
• complete (allow anything to be able to add metrics)
• incremental (dont add everything just yet, add as we need)

I have one proposal with two possible implementation approaches. The idea is to make one module that has metrics stored at particular paths, so things like /node/gateway/requests/total would be a growing int, but namespaced under /node/gateway/requests/, with other things like /node/gateway/requests/per-second or something.

From an interfaces standpoint, the idea i have would be to make one Metrics or MetricsCollector object that is either passed down or available globally like the go loggers (the 2 approaches). And allow different go packages to use the same (or derived) metrics object. Concretely, is should be able to instantiate multiple IPFS Nodes in the same process (or a bssim and many ipfs nodes) and have them all log all their metrics to the same object, under different namespaces.

(So, instead of having a package implement its own collection and expose functions to get the metrics, the package is given a metrics object (global or passed in), that they write their metrics to.)

IIRC, https://github.com/ipfs/go-ipfs/pull/1163/files does some namespacing of metrics.

[jbenet]

Some of the metrics bitswap-ml may need: ipfs-inactive/bitswap-ml#1

[whyrusleeping]

I had some plans to write a 'bs-top' that would be a process monitor for bitswap. It would essentially register for notifications from bitswap, and be notified of what each worker is doing as soon as it starts doing it. That way, we could have a nice interface that shows live information about what bitswap is doing.

Although, that may be tangental to the metrics we're discussing here.

[jbenet]

@whyrusleeping sweet! i want it to. we should make these "dashboards" for different subsystems (dht, bitswap, repo, api, gateway, etc) (top is a dashboard)

[whyrusleeping]

var clones []Person
for i := 0; i < 50; i++ {
    clones = append(clones, whyrusleeping.Clone())
}

[whyrusleeping]

maybe we want to discuss a general strategy for dashboards somewhere? We could either do them through notifications, which might be obnoxious, or we could do it through our new eventlogs API. and just have bitswap emit eventlogs for the dashboard

[jbenet]

@Heems @lgierth let's maybe list metrics here we want to add, and check them as they're added, like so:

• number of peers connected
• total bytes transmitted per peer