Metric workflow_task_execution_failed is used with different sets of labels causing errors with Prometheus

[wsny]

Expected Behavior

Should not receive errors about metrics being registered incorrectly.

Actual Behavior

Receiving a Prometheus error about the workflow_task_execution_failed metric being registered with different labels.

a previously registered descriptor with the same fully-qualified name as Desc{fqName: "temporal_workflow_task_execution_failed_total", help: "temporal_workflow_task_execution_failed_total counter", constLabels: {}, variableLabels: {app,client_name,namespace,environment,workflow_type,failure_reason,task_queue,worker_type,deployment,activity_type,env}} has different label names or a different help string

Steps to Reproduce the Problem

I am still working on STR's (don't know how to break all the below with test code yet), but I can see that this metric is being used with different tags in different places.

#1295 added a tag here

sdk-go/internal/internal_task_pollers.go

Line 421 in 69bc6c3

metricsHandler.WithTags(metrics.WorkflowTaskFailedTags(failureReason)).Counter(metrics.WorkflowTaskExecutionFailureCounter).Inc(1)

But it is not being added in these uses of the metric.

sdk-go/internal/internal_event_handlers.go

Line 1108 in 69bc6c3

weh.metricsHandler.Counter(metrics.WorkflowTaskExecutionFailureCounter).Inc(1)

sdk-go/internal/internal_event_handlers.go

Line 1298 in 69bc6c3

weh.metricsHandler.Counter(metrics.WorkflowTaskExecutionFailureCounter).Inc(1)

And that will cause issues with Prometheus, as shown in this example.

package main

import (
	"fmt"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/uber-go/tally/v4"
	tallyprom "github.com/uber-go/tally/v4/prometheus"

	"go.temporal.io/sdk/client"
	sdktally "go.temporal.io/sdk/contrib/tally"
)

func main() {
	metricsHandler := setupPrometheus()
	// These lines in any order will cause an error where the metrics have different labels
	metricsHandler.Counter("testing").Inc(1)
	metricsHandler.WithTags(map[string]string{"someName": "someValue"}).Counter("testing").Inc(1)
}

// Just setup stuff for Prometheus below here
var PrometheusSanitizeOptions = tally.SanitizeOptions{
	NameCharacters:       tally.ValidCharacters{Ranges: tally.AlphanumericRange, Characters: []rune{'_'}},
	KeyCharacters:        tally.ValidCharacters{Ranges: tally.AlphanumericRange, Characters: []rune{'_'}},
	ValueCharacters:      tally.ValidCharacters{Ranges: tally.AlphanumericRange, Characters: []rune{'_'}},
	ReplacementCharacter: tally.DefaultReplacementCharacter,
}

func setupPrometheus() client.MetricsHandler {
	cfg := tallyprom.Configuration{
		ListenAddress: "localhost:9090",
	}
	reporter, _ := cfg.NewReporter(tallyprom.ConfigurationOptions{
		Registry: prometheus.NewRegistry(),
		OnError: func(err error) {
			fmt.Printf("BOOM: %+v\n", err)
		},
	})
	scopeOpts := tally.ScopeOptions{CachedReporter: reporter, SanitizeOptions: &PrometheusSanitizeOptions}
	scope, _ := tally.NewRootScope(scopeOpts, time.Second)
	scope = sdktally.NewPrometheusNamingScope(scope)
	return sdktally.NewMetricsHandler(scope)
}

Which causes this error

a previously registered descriptor with the same fully-qualified name as Desc{fqName: "testing_total", help: "testing_total counter", constLabels: {}, variableLabels: {someName}} has different label names or a different help string

Specifications

• Version: sdk-go@v1.26.1

[cretz]

👍 We should ideally only be recording this metric in one place and it should always contain the same tag/label set. Thanks for reporting! This is on our backlog and we will see about fixing.

[AmirSoleimani]

I noticed that the issue has been closed and addressed. I've been facing the same issue and the merged changes addressed it (I had to pull the master branch to test). However, I haven't seen a new release with the fix yet. Can you provide any updates on when the fix might be included in a release? Thank you for your help! - cc @yuandrew

[yuandrew]

Hey Amir, sorry for the delay, there are a few more changes we are hoping to get into the next release, I am hoping we can cut a new release in the next 2 weeks!

[yuandrew]

v1.30.0 has just been released