feat: add retry count value of custom metric

[moonyoungCHAE]

Fixes #7705

Motivation

Currently, there is no way to obtain the retry count for each step. If we can use the retry count as a custom metric, it would be useful for identifying problematic steps.

Modifications

I have added the retry count as a metric value.

Verification

I conducted a test as follows:
If a step is successful, the retry metric will not be shown. However, if a step fails and is retried, the retry metric should be displayed.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: retry-with-steps-
spec:
  entrypoint: retry-with-steps
  templates:
    - name: retry-with-steps
      steps:
        - - name: success
            template: random-success
          - name: fail
            template: random-fail
    - name: random-success
      container:
        image: python:alpine3.6
      metrics:
        prometheus:
          - name: retry_count_success
            help: "retry count of success step that would not be appear, as it would not be retried"
            counter:
              value: "{{retries}}"
    - name: random-fail
      retryStrategy:
        limit: "3"
      container:
        image: python:alpine3.6
        command: [python, -c]
        args: ["invalid_args@@@; import sys; print('retries: {{retries}}'); exit_code = random.choice([0, 1, 1]); sys.exit(exit_code)"]
      metrics:
        prometheus:
          - name: retry_count_fail
            help: "retry count of fail and retry step that would be appear, as it would be retried"
            counter:
              value: "{{retries}}"

The fail step retried 3 times.

And this is what I got from localhost:9090/metrics

# HELP argo_workflows_retry_count_fail retry count of fail and retry step that would be appear, as it would be retried
# TYPE argo_workflows_retry_count_fail counter
argo_workflows_retry_count_fail 3

[juliev0]

So sorry, I'm going through old PRs and I see that unfortunately this one has been forgotten. I will try to review it soon.

Argo Workflows needs more people approving PRs unfortunately. If your company is by chance interested, I am gauging interest in this idea: #12229