feat(metrics) Limit Cardinality of CSV metrics

[awgreene]

This commit introduces a change that limits the number of metrics that
an OLM cluster reports at any given time for a CSV.

The first metric introduced is called csv_succeeded, which tracks CSVs that
have reached the succeeded phase. The following information is
provided about the CSV via labels: name, version. The value of this
metric will always be 0 or 1.

The second metric introduced is called csv_abnormal, which is reported
whenever the CSV is updated and has not reached the succeeded phase. The
following information is provided about the CSV via labels: name,
version, phase, reason. Whenever a CSV is updated, the existing
timeseries is deleted and replaced by an updated version.

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Docs updated or added to /docs
• Commit messages sensible and descriptive

[awgreene]

Example Cases:

When a CSV is updated but not in the succeeded phase

# HELP csv_abnormal Successful CSV Install
# TYPE csv_abnormal gauge
csv_abnormal{name="etcd-operator.v1.0.0",phase="Failed",reason="UnsupportedOperatorGroup",version="1.0.0"} 1.0
...
...
...
# HELP csv_succeeded Successful CSV Install
# TYPE csv_succeeded gauge
csv_succeeded{name="packageserver",version="1.0.0"} 1.0
csv_succeeded{name="etcd-operator.v1.0.0",version="1.0.0"} 0.0

Note: When a CSV is updated, the old metric is deleted.

When a CSV reaches the succeeded phase

# HELP csv_succeeded Successful CSV Install
# TYPE csv_succeeded gauge
csv_succeeded{name="packageserver",version="1.0.0"} 1.0
csv_succeeded{name="etcd-operator.v1.0.0",version="1.0.0"} 1.0

Note: Notice that the csv_abnormal timeseries was removed

[awgreene]

@ecordell it should be noted that this data will get messy if multiple instances of the same operator are installed.

[awgreene]

/test e2e-gcp-upgrade

[awgreene]

/retest

[ecordell]

/lgtm

[ecordell]

/hold

while I spin up a cluster to look at this :)

[awgreene]

/hold

while I spin up a cluster to look at this :)

Sure - let me know if you find anything interesting!

[ecordell]

Checking CSV succeeded:
blue is packageserver, which is always good
green is etcd-operator - I killed its deployment to watch it go unhealthy

This is csv_abnormal - I just see it for installready, when olm detected the deployment was deleted and reconciles from the CSV.

This looks good, but I did expect to see more values for "abnormal". I think what's happening is that we're deleting the abnormal states from prometheus before prometheus can scrape them.

Knowing when to delete them is tricky, since the scrape time is configurable. I think it's 20s by default. We may need to do something where we keep the "old" timeseries around for a while before deleting them.

[kevinrizza]

Checking CSV succeeded:
blue is packageserver, which is always good
green is etcd-operator - I killed its deployment to watch it go unhealthy

This is csv_abnormal - I just see it for installready, when olm detected the deployment was deleted and reconciles from the CSV.

This looks good, but I did expect to see more values for "abnormal". I think what's happening is that we're deleting the abnormal states from prometheus before prometheus can scrape them.

Knowing when to delete them is tricky, since the scrape time is configurable. I think it's 20s by default. We may need to do something where we keep the "old" timeseries around for a while before deleting them.

@ecordell Maybe this is a silly question, but what value do we get from seeing every step here rather than just seeing "it's in a good state" or "it's not in a good state" ? Is there some unique knowledge we would gain from seeing all of those steps? I was thinking that we care more about "what step are we stuck in" in the same way that I can learn that by kubectl getting the CSV

My concern is that being aware of the state of these things and deleting them later seems like it adds a medium amount of complexity to managing the metric endpoint's state for the value that it adds.

[ecordell]

That's a good point, and I do think it would be reasonable to merge this as-is and accept that not all states will necessarily get picked up by prometheus.

It would be nice to see the full lifecycle of operators reported if possible though. Perhaps we do that as a follow up.

[ecordell]

/lgtm
/hold cancel

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awgreene, ecordell

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ecordell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ecordell]

/retest

[awgreene]

/retest

[awgreene]

/retest