It occurs "collected metric %s %s was collected before with the same name and label values" error in the lastest version

[nature502]

We use prometheus to monitor our golang application. It works well. When we upgrade to release 0.8.0 (commit ID c5b7fcc). It occurs "collected metric %s %s was collected before with the same name and label values" error. I find the old version ignores consistency checks, but the lastest must do it . So, when we use counter to count, it occurs errors.
Code in previous:
// EnableCollectChecks enables (or disables) additional consistency checks
// during metrics collection. These additional checks are not enabled by default
// because they inflict a performance penalty and the errors they check for can
// only happen if the used Metric and Collector types have internal programming
// errors. It can be helpful to enable these checks while working with custom
// Collectors or Metrics whose correctness is not well established yet.
func EnableCollectChecks(b bool) {
defRegistry.collectChecksEnabled = b
}

// decide skip or not.
if r.collectChecksEnabled {
if err := r.checkConsistency(metricFamily, dtoMetric, desc, metricHashes); err != nil {
return 0, err
}
}

Code in lastest
// Gather.
for metric := range metricChan {
.................
metricFamily, ok := metricFamiliesByName[desc.fqName]
if ok {
.............................
} else {
.......................................
if err := checkMetricConsistency(metricFamily, dtoMetric, metricHashes, dimHashes); err != nil {
errs = append(errs, err
continue
}

I think problem is here. It ignores locking of metricFamiliesByName in concurrent environment.

[beorn7]

I don't see the problem. metricFamiliesByName is never used in a concurrent context. It's a local variable, and only used in the for loop where only one goroutine is acting on it.

Are you perhaps really collecting inconsistent metrics? Could you provide more detail on which metric the error message is triggered?

You also might have run into a hash collision, see #222 . That would be super interesting to hunt down.

[nature502]

We use counter to count url hits. We have two urls. One is "/api/v1/app", the other is "/api/v1/ad". Below is the code in our application.

    webUrlHitCounts = prom.NewCounterVec(
        prom.CounterOpts{
            Namespace: "xxx",
            Subsystem: "web",
            Name:      "url_hit_counts",
            Help:      "xxx web url hit counts.",
        },
        []string{labelUrl}, // labelUrl = "url_path"
    )
       webUrlHitCounts.WithLabelValues(ctx.PathString()).Inc()

It occurs the below error in page.
url http://xxx.xxx.xxx.xxx:xxxx/_/metrics
**An error has occurred during metrics collection:

9 error(s) occurred:

• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/adp" > counter:<value:6 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/adp" > counter:<value:3 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ack" > counter:<value:2 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ack" > counter:<value:5 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ack" > counter:<value:3 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ac" > counter:<value:49 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ad" > counter:<value:6 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/ac" > counter:<value:9 > was collected before with the same name and label values
• collected metric xxx_web_url_hit_counts label:<name:"url_path" value:"/api/v1/app" > counter:<value:10091 > was collected before with the same name and label values.**

In addition, I find a very puzzled thing. We only count two urls, one is "/api/v1/app", the other is "api/v1/ad". The result shows others urls which we do not have. It occurs in previous and lastest version.

[beorn7]

In addition, I find a very puzzled thing. We only count two urls, one is "/api/v1/app", the other is "api/v1/ad". The result shows others urls which we do not have. It occurs in previous and lastest version.

This is a hint that you have some rogue collector in your code that feeds into the same metric. Only with v0.8.0, you would notice that. If you are really only using a stock CounterVec, the consistency check should never result in an error (unless there is a bug in the library or you have really run into a hash collision).

[beorn7]

I assume this is clarified. If not, please re-open with more evidence.

[detailyang]

@beorn7 Same issue here, it's hard to debug it:(. I cannot construct the issue in the manual. Do you have any idea?

Many thanks

[beorn7]

@detailyang This issue was filed for a change introduced in v0.8 more than two years ago. It is unlikely that you have run into the same issue now.

You might have discovered an entirely new bug. In that case, you should file a new issue, providing the necessary context so that others can find the bug. (Ideally, you'll find it yourself.)

However, it seems you don't know yet why the error message shows up and you have questions how to understand why it is showing up. It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. (A closed GitHub issue is even less suited.) On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[detailyang]

@beorn7

I got an idea to get more information about this. It's hard to reproduce the strange issue but I will look at more. I will join the malling list to solve this with the whole community.

Anyway, many thanks:)

[ainiml]

Getting the same error

* collected metric "atIfIndex" { label:<name:"atIfIndex" value:"95" > label:<name:"atNetAddress" value:"1.10.10.10" > gauge:<value:95 > } was collected before with the same name and label values

[alexellis]

I ran into this because of:

	for _, totals := range queuedJobs {
		c.queuedJobsGauge.WithLabelValues(totals.Owner).Set(float64(totals.QueuedTotal))
	        c.queuedJobsGauge.Collect(ch)
	}

And it should have been:

	for _, totals := range queuedJobs {
		c.queuedJobsGauge.WithLabelValues(totals.Owner).Set(float64(totals.QueuedTotal))
	}

	c.queuedJobsGauge.Collect(ch)

Easy mistake to make and very hard to figure out.

[Nachtfalkeaw]

Hello,
I maybe have the same error with some pfsense MIBs and atest snmp_exporter 0.25.0 and its generator:

An error has occurred while serving metrics:

3 error(s) occurred:

• collected metric "begemotIfMaxspeed" { counter:{value:0}} was collected before with the same name and label values
• collected metric "begemotIfMaxspeed" { counter:{value:1e+09}} was collected before with the same name and label values
• collected metric "begemotIfMaxspeed" { counter:{value:100}} was collected before with the same name and label values

I try to walk these:
walk: [sysUpTime,pfInterfaces,pfTablesTblNumber,pfTablesTblTable,pfCounter,pfStateTable,pfStatus,freeBSDsrc,begemotIp,ipv4InterfaceTableLastChange,ipv4InterfaceTable,ipv4InterfaceEntry,ipv4InterfaceIfIndex,ipTrafficStats,ipIfStatsTable,ipMIBConformance]

[mihail641]

@nature502 , @detailyang , @ainiml , @Nachtfalkeaw , If you using Fiber, in Fiber http parameters, body - unsafe and mutable. Set Immutable: true in fiber.Config.
Or you can copy unsafe string value in immutable string fmt.Sprinf("%s", myVal) for label values.