feat: allow custom labels for count metrics. Fixes #9849

[LilTwo]

Fixes #9849

Motivation

Since all custom non-realtime metrics are emitted after workflow completed, and realtime metrics are emitted after workflow start to run.

Currently there is now way to know pending workflow count grouped by labels, name or others

Use case:
We use different semaphore configs for multiple kinds of workflows and need to have alerts to monitor if certain kind of workflows are blocked

Modifications

Allow users to add custom labels to arog_workflow_count
while keeping the metrics unchanged if new added fields are not presented

example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
    test: "123"
  annotations:
    workflows.argoproj.io/description: |
      This is a simple hello world example.
spec:
  entrypoint: whalesay
  metrics:
    customLabels:
      count:
        key: "{{workflow.labels.test}}"
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

resulting metrics
argo_workflows_count_new{key="123",status="Succeeded"} 1

Verification

Tests will be added after the design is confirmed

[agilgur5]

Fixes #9839

I think you meant #9849 😅 your title has it!

[LilTwo]

yes you're right, thanks😅

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[juliev0]

So sorry, I'm going through old PRs and I see that unfortunately this one has been forgotten. I will try to review it soon.

Argo Workflows needs more people approving PRs unfortunately. If your company is by chance interested, I am gauging interest in this idea.

[juliev0]

@LilTwo do you still want this one reviewed? I see the Stale bot marked it as "stale", which we recently removed since that was clearly unfairly marking PRs stale when somebody was waiting for them to be reviewed. I can remove that label if you like. Thanks for your patience!

[terrytangyuan]

Closing since this has been inactive for several months. Feel free to re-open if you pick it up again.

[linkingli]

is there any reason why should we close this pr? i would like to have this fearture

[tooptoop4]

where @LilTwo