tekton_pipelines_controller_pipelinerun_count metric counter increases without having any pipeline executed

[gmeghnag]

Expected Behavior

To have the tekton_pipelines_controller_pipelinerun_count metric counter increased only when new pipelines are executed.

Actual Behavior

After a few hours the counter is increased without having any pipelines executed.

Steps to Reproduce the Problem on OpenShift

1. Run at least one pipeline in the cluster
2. Expose the metrics endpoint:

$ oc expose -n openshift-pipelines svc/tekton-pipelines-controller --port=9090 --path="/metrics"

3. Check the counter value:

$ METRICS_ENDPOINT=$(oc get route tekton-pipelines-controller -n openshift-pipelines -o jsonpath="{.spec.host}")/metrics
$ curl -s -k $METRICS_ENDPOINT | grep -v "#" | grep tekton_pipelines_controller_pipelinerun_count

4. Wait a few hours (4/5) without executing any pipelines, and after re-checking the counter value, you will see it increased

Additional Info

• Kubernetes version:

Kubernetes Version: v1.20.0+9689d22

• Tekton Pipeline version:

$ tkn version
Client version: 0.13.1
Pipeline version: v0.22.0
Triggers version: v0.12.1

[gmeghnag]

DurationAndCount method:
https://github.com/tektoncd/pipeline/blob/release-v0.22.x/pkg/reconciler/pipelinerun/metrics.go#L126-L164

Where is Invoked:
https://github.com/tektoncd/pipeline/blob/release-v0.22.x/pkg/reconciler/pipelinerun/pipelinerun.go#L199-L206

[guillaumerose]

Yes I confirm this behaviour.

If I install Tekton with a resyncPeriod of the controller = 10 seconds, then when I create a single pipeline, every 10s, the counter tekton_pipelines_controller_pipelinerun_count{status="success"} is increased by 1.

Looking at the code, it shows also the duration histogram is incorrect. Older pipelines are counted many more times than younger pipelines.

The bug is by design and I think, the only way to remove it is to refactor the code to create this counter only using the lister and not using the controller loop.
If we use the controller loop, we need to use the beforeCondition/afterCondition check to find out if the pipeline was already counted or not. .. But it's not possible: if the controller restarts, we loose this information and the counter will start again at 0.

[khrm]

@wlynch I fixed this in a pr #4468.

@guillaumerose We generally use the rate function of Prometheus query to avoid instance restart related issues. Even with lister loop, information will be lost during restart because pipelinerun will be deleted by the end-user. So I think before and after condition check is sufficient to resolve this.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[lbernick]

/remove-lifecycle stale