collected metric process_network_receive_bytes_total counter with unregistered descriptor

[geberl]

I just updated the library version from v1.19.1 -> v1.20.0.

I now can't access my service's metrics any more.

curl localhost:8888/metrics

An error has occurred while serving metrics:

2 error(s) occurred:
* collected metric process_network_receive_bytes_total counter:{value:456312} with unregistered descriptor Desc{fqName: "process_network_receive_bytes_total", help: "Number of bytes received by the process over the network.", constLabels: {}, variableLabels: {}}
* collected metric process_network_transmit_bytes_total counter:{value:387314} with unregistered descriptor Desc{fqName: "process_network_transmit_bytes_total", help: "Number of bytes sent by the process over the network.", constLabels: {}, variableLabels: {}}

Might be in connection with #1555 ?

Is anyone else seeing this?

Revert to v1.19.1 fixed it for me.

[CubicrootXYZ]

Encountered the same issue when upgrading to v1.20.0.

[geberl]

Looked a bit deeper, we use reg := prometheus.NewPedanticRegistry(), not reg := prometheus.NewRegistry().

client_golang/prometheus/registry.go

Line 85 in 989a6d0

func NewPedanticRegistry() *Registry {

[zetaab]

we are hitting this same issue

Got 500 status code

2 error(s) occurred:

• collected metric process_network_receive_bytes_total counter:{value:92659} with unregistered descriptor Desc{fqName: "process_network_receive_bytes_total", help: "Number of bytes received by the process over the network.", constLabels: {}, variableLabels: {}}
• collected metric process_network_transmit_bytes_total counter:{value:125661} with unregistered descriptor Desc{fqName: "process_network_transmit_bytes_total", help: "Number of bytes sent by the process over the network.", constLabels: {}, variableLabels: {}}

we use also prometheus.NewPedanticRegistry(). This issue happens only with linux + amd64 setup. I cannot reproduce it with my arm mac.

[zetaab]

I can confirm that the issue is NewPedanticRegistry. I tested here elisasre/go-common#348 with and without it. I think #1555 is the breaking PR

[timofurrer]

We are also using the NewPedanticRegistry and running into the same issue. This is a breaking change for us coming from v1.19.1.

[zetaab]

cc @bwplotka @ArthurSens @kakkoyun version v1.20.0 contains breaking change which was not documented at all. Currently the version v1.20.0 is broken if someone uses linux and NewPedanticRegistry.

[bwplotka]

Thanks! We will fix promptly and make sure we have the process to not allow that in future.

[bwplotka]

Traveling, but quickly checking: Bug is that Describe method does not have those new desc items:

client_golang/prometheus/process_collector.go

Line 135 in 989a6d0

func (c *processCollector) Describe(ch chan<- *Desc) {

To ensure we don't regress more, let's add tests against pedantic registry for all native collectors. Cc @ArthurSens

[ArthurSens]

I've created #1587 with 2 commits:

• First one uses NewPedanticRegistry in process collector tests.
• Second updates the Describe() method adding the new metrics.

Personally, I cannot reproduce this error locally and I've added the first commit hoping that at least the CI would catch it. To my surprise, even in CI the test pass 🤔

Could someone who can reproduce the bug, try the branch from #1587 ?

[zetaab]

@ArthurSens it does fix the issue. Commit elisasre/go-common@827df1e contains v1.20.0 (the CI will fail). And then trying PR #1587 elisasre/go-common@a3ab72b the PR builds are going through

[ArthurSens]

Could you clarify what OS is being used in your CI? https://github.com/elisasre/go-common/blob/76b822e4600d7b6b9a8dec22b1eb62f34fd1ced1/.github/workflows/default.yaml#L14

[zetaab]

@ArthurSens ubuntu 20.04 and amd64

[ArthurSens]

That's strange, we're also running ubuntu. Ideally we should find a failing test before saying that this fix the issue for sure 😭

[geberl]

Working for me in v1.20.1 🎉
I'm closing this issue now. Thanks for your work @ArthurSens !