Add new Prometheus alerts and label existing alerts #2998

arnongilboa · 2023-11-22T14:24:57Z

What this PR does / why we need it:

CDINoDefaultStorageClass - not having a default (or virt default) SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert.
CDIDefaultStorageClassDegraded - when the default (or virt default) SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration).
CDIStorageProfilesIncomplete - add storageClass and provisioner labels.
CDIDataImportCronOutdated - add dataImportCron namespace and name labels.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

Add Prometheus alerts CDINoDefaultStorageClass and CDIDefaultStorageClassDegraded; add informative labels to existing alerts CDIStorageProfilesIncomplete and CDIDataImportCronOutdated

doc/metrics.md

akalenyu

Just summing up the offline discussion

akalenyu · 2023-11-23T16:33:04Z

cmd/cdi-controller/controller.go

 	metrics.Registry.MustRegister(controller.DefaultVirtStorageClassesGauge)
+	metrics.Registry.MustRegister(controller.DefaultStorageClassWithSmartCloneAndRWXGauge)


Just following up on offline discussion:
Maybe make DefaultStorageClassWithSmartCloneAndRWXGauge just about virt storage classes? Even then I fear we can't bug people who have some basic storage solution. But as long as we see this in telemetry we can adjust ourselves

I tend to agree.

akalenyu · 2023-11-23T16:34:41Z

pkg/operator/controller/prometheus.go

@@ -221,6 +221,21 @@ func getAlertRules(runbookURLTemplate string) []promv1.Rule {
 				componentAlertLabelKey:    componentAlertLabelValue,
 			},
 		),
+		generateAlertRule(
+			"CDINoDefaultStorageClass",
+			"kubevirt_cdi_default_storageclasses == 0 and kubevirt_cdi_default_virt_storageclasses == 0 and kubevirt_cdi_datavolumes_pending_for_default_storageclass > 0",


Maybe this should also just be about virt storage classes? DataImportCrons without explicit SC should all work if a virt storage class is defined

Not sure I get you. User can define default and/or virt default SC, so why require the virt one specifically?

Just a similar suggestion to #2998 (comment). Reading it again though I think it's fine since it'll only alert if all of them are 0, so virt sc existence will bail us out of noise.

akalenyu · 2023-11-23T16:55:01Z

pkg/controller/storageprofile-controller.go

 	DefaultVirtStorageClassesGauge.Set(float64(defaultVirtStorageClassCount))

+	hasSmartCloneAndRWX := 0
+	if defaultStorageClassCount > 0 || defaultVirtStorageClassCount > 0 {
+		sc := cc.GetFallbackStorageClass(storageClassList, cdiv1.DataVolumeKubeVirt)


I think you're already (unintentionally?) always looking up the virt default storage class
(always pass cdiv1.DataVolumeKubeVirt content type)

Your fallback func prefers the virt default storage class, but returns the default SC if virt default is not set.

I am saying that you already unintentionally implemented #2998 (comment) since this cc.GetFallbackStorageClass(storageClassList, cdiv1.DataVolumeKubeVirt) will only bring you virt storage classes

avlitman · 2023-11-26T13:39:25Z

Hi @arnongilboa and @akalenyu I'm also working on the same files to refactor the monitoring code to use external pkg we created, and I think it will be better if I work on top of this pr.
The question is, do you think this pr will be ready soon, or is it still WIP?

I will upload an initial pr not including this changes by the end of today, still facing compile issues...
Thanks!

avlitman · 2023-11-26T14:01:40Z

pkg/monitoring/metrics.go

 	DefaultVirtClasses: {
 		Name: "kubevirt_cdi_default_virt_storageclasses",
 		Help: "Number of default virt storage classes currently configured",
 		Type: "Gauge",
 	},
+	DataVolumesPendingForDefaultStorageClass: {
+		Name: "kubevirt_cdi_datavolumes_pending_for_default_storageclass",


Since there is other metrics in the form of kubevirt_cdi_***_storage***
maybe it's better to name this one kubevirt_cdi_default_storageclass_pending_datavolumes

Will be renamed. Thanks Aviv!

arnongilboa · 2023-11-26T14:06:56Z

Hi @arnongilboa and @akalenyu I'm also working on the same files to refactor the monitoring code to use external pkg we created, and I think it will be better if I work on top of this pr. The question is, do you think this pr will be ready soon, or is it still WIP?

I will upload an initial pr not including this changes by the end of today, still facing compile issues... Thanks!

It will be ready during this week after another review by @sradco, so you better wait with your PR until this one is in.

avlitman · 2023-11-26T14:09:30Z

pkg/monitoring/metrics.go

+		Type: "Gauge",
+	},
+	DefaultStorageClassWithSmartCloneAndRWX: {
+		Name: "kubevirt_cdi_default_storageclass_with_smart_clone_and_rwx",


To match kubevirt_cdi_dataimportcron_outdated
Maybe kubevirt_cdi_default_storageclass_smart_clone_and_rwx

Also can you explain the "smart" part, e.g. kubevirt_cdi_clone_pods_high_restart are also smart clone pods?

Removed this metric, using labels on the kubevirt_cdi_storageprofile_status metric instead.

akalenyu · 2023-11-28T18:20:58Z

tests/monitoring_test.go

+			if f.SnapshotSCName != defaultStorageClass.Name && f.CsiCloneSCName != defaultStorageClass.Name {
+				Skip("Default storage class does not support snapshot or CSI clone")
+			}
+			if !f.IsSnapshotStorageClassAvailable() && !f.IsCSIVolumeCloneStorageClassAvailable() {


I think this test will always get skipped since the destructive lane only has HPP (no snapshot support)
Maybe since we don't actually need snapshot functionality, just the vsc object, we can set it up ad hoc just for this test?

Right, I'll try your nice idea :)

akalenyu · 2023-11-28T18:22:19Z

tests/monitoring_test.go

+			By("Restore storage profile")
+			updateDefaultStorageClassProfileClaimPropertySets(nil)
+
+			By("Ensure metric values after restoring storage profile")


I was about to suggest to ensure that the metric is restored after the cluster state is "fixed"
but I see you already do that in all of the tests. Cool
Testing that alerts go away when things are good is really critical from my experience

arnongilboa · 2023-12-04T12:50:55Z

/retest

sradco · 2023-12-05T09:41:43Z

pkg/operator/controller/prometheus.go

+			},
+			map[string]string{
+				severityAlertLabelKey:     "warning",
+				healthImpactAlertLabelKey: "warning",


@arnongilboa Please make sure this is the correct impact level on the operator health.

sradco · 2023-12-05T09:41:48Z

pkg/operator/controller/prometheus.go

+			},
+			map[string]string{
+				severityAlertLabelKey:     "warning",
+				healthImpactAlertLabelKey: "warning",


@arnongilboa Please make sure this is the correct impact level on the operator health.

ShellyKa13 · 2023-12-05T13:46:45Z

pkg/controller/datavolume/controller-base.go

+	httpClient *http.Client
+
+	// DataVolumePendingGauge is the metric we use to count the DataVolumes pending for default storage class to be configured
+	DataVolumePendingGauge = prometheus.NewGauge(


I wish this variable and all metrics related functions would be in a dedicated package

Each metric is maintained in the relevant packge where the logic is (DV, DIC, StorageProfile etc.). I don't think we should put them all in one package. However, the alerts are based on multiple metrics and therefore defined in a single package.

ShellyKa13 · 2023-12-05T13:48:17Z

pkg/controller/datavolume/controller-base.go

@@ -388,6 +402,25 @@ func getSourceRefOp(log logr.Logger, dv *cdiv1.DataVolume, client client.Client)
 	}
 }

+func updatePendingDataVolumesGauge(c client.Client, dv *cdiv1.DataVolume) {


I saw you changed the client name from c to client in another PR so maybe you should do it here too :)

ShellyKa13 · 2023-12-05T13:49:57Z

pkg/controller/datavolume/controller-base.go

+	}
+
+	dvList := &cdiv1.DataVolumeList{}
+	if c.List(context.TODO(), dvList, client.MatchingFields{dvPhaseField: string(cdiv1.Pending)}) != nil {


do we want to log (in a low verbosity) that we failed getting the list and that why the mertic will not get updated?

ShellyKa13 · 2023-12-05T13:51:41Z

pkg/controller/datavolume/controller-base.go

+			dvCount++
+		}
+	}
+	DataVolumePendingGauge.Set(float64(dvCount))


Im not so familiar with how the metric is being calculated but is there any considerations to namespaces at all?

We are interested in the total default-SC pending DVs in all namespaces (whether it is zero or not).

ShellyKa13 · 2023-12-05T13:56:31Z

pkg/controller/storageprofile-controller.go

-	}
-
-	return reconcile.Result{}, r.computeMetrics()
+	return r.reconcileStorageProfile(storageClass)


this way we will compute the metric only if no error occured are we sure thats what we want?

We update the metric of a specific SC only if no error occured, which seems to be a consistent behavior. In case of SC deletion, we delete the metric.

ShellyKa13 · 2023-12-07T10:30:01Z

pkg/controller/storageprofile-controller.go

-	// DefaultVirtStorageClassesGauge is the metric we use to alert about multiple default virt storage classes
-	DefaultVirtStorageClassesGauge = prometheus.NewGauge(
+	// StorageProfileStatusGaugeVec is the metric we use to alert about storage profile status
+	StorageProfileStatusGaugeVec = prometheus.NewGaugeVec(


As mentioned I think it will be much cleaner to have all Prometheus functionality and variables in dedicated pkg/file.

Not sure I agree, as already answered. However we can do it in a follow-up PR.

Hi @ShellyKa13, the metrics and alerts would be moved to a dedicated package as part of this PR #3009

ShellyKa13 · 2023-12-07T10:56:48Z

pkg/operator/controller/prometheus.go

+			"CDINoDefaultStorageClass",
+			`sum(kubevirt_cdi_storageprofile_status{default="true"} or on() vector(0)) +
+			sum(kubevirt_cdi_storageprofile_status{virtdefault="true"} or on() vector(0)) +
+			(count(kubevirt_cdi_datavolume_pending == 0) or on() vector(0)) == 0`,


shouldnt the count of datavolumes pending needs to be bigger then 0?

we want kubevirt_cdi_datavolume_pending > 0, so we check it here by count(kubevirt_cdi_datavolume_pending == 0) == 0

but then dont you want to count the (pending == 1) > 0?
wouldnt count(kubevirt_cdi_datavolume_pending == 0) == 0 count there are no not pending dv?

The only way I know (discussing it with PromQL expert @machadovilaca !) to do logical && in such promQL expr is by using sum (+) of the parts which we want to be 0 and compare the sum to 0.

- CDINoDefaultStorageClass - not having a default (or virt default) SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert. - CDIDefaultStorageClassDegraded - when the default (or virt default) SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration). - CDIStorageProfilesIncomplete - add storageClass and provisioner labels. - CDIDataImportCronOutdated - add dataImportCron namespace and name labels. Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Including the VolumeSnapshot/Class/Content crds for the CDIDefaultStorageClassDegraded alert func test. Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

sradco · 2023-12-12T12:19:16Z

doc/metrics.md

 ### kubevirt_cdi_operator_up
 CDI operator status. Type: Gauge.
+### kubevirt_cdi_storageprofile_status


@arnongilboa Please elaborate on this metric. I see from the alert that there are 3 labels: default and virtdefault, complete. What do they mean? are there additional labels?

kubevirt-bot · 2023-12-18T09:42:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akalenyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [akalenyu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

arnongilboa · 2023-12-18T22:01:07Z

/retest

ShellyKa13 · 2023-12-19T11:01:08Z

/ lgtm

sradco · 2023-12-19T11:27:49Z

/lgtm

Manual backport of kubevirt#2998 & kubevirt#3038 - CDINoDefaultStorageClass - not having a default (or virt default) SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert. - CDIDefaultStorageClassDegraded - when the default (or virt default) SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration). - CDIStorageProfilesIncomplete - add storageClass and provisioner labels. - CDIDataImportCronOutdated - add dataImportCron namespace and name labels. Also: * Rename the metric kubevirt_cdi_storageprofile_status to kubevirt_cdi_storageprofile_info since it always reports value 1, where the label values provide the details about the storage class and storage profile. * Add snapshot manifests for tests and deploy snapshot CRDs in the hpp destructive lane Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

…3040) Manual backport of #2998 & #3038 - CDINoDefaultStorageClass - not having a default (or virt default) SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert. - CDIDefaultStorageClassDegraded - when the default (or virt default) SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration). - CDIStorageProfilesIncomplete - add storageClass and provisioner labels. - CDIDataImportCronOutdated - add dataImportCron namespace and name labels. Also: * Rename the metric kubevirt_cdi_storageprofile_status to kubevirt_cdi_storageprofile_info since it always reports value 1, where the label values provide the details about the storage class and storage profile. * Add snapshot manifests for tests and deploy snapshot CRDs in the hpp destructive lane Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Manual backport of kubevirt#2998 & kubevirt#3038 - CDINoDefaultStorageClass - not having a default SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert. - CDIDefaultStorageClassDegraded - when the default SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration). - CDIStorageProfilesIncomplete - add storageClass and provisioner labels. - CDIDataImportCronOutdated - add dataImportCron namespace and name labels. Also: * Rename the metric kubevirt_cdi_storageprofile_status to kubevirt_cdi_storageprofile_info since it always reports value 1, where the label values provide the details about the storage class and storage profile. * Add snapshot manifests for tests and deploy snapshot CRDs in the hpp destructive lane Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

…3041) Manual backport of #2998 & #3038 - CDINoDefaultStorageClass - not having a default SC is surely not an OpenShift error, as admins may prefer their cluster users to only use explicit SC names. However, in the CDI context when DV is created with default SC but default does not exist, we will fire an error event and the PVC will be Pending for the default SC, so when there are such Pending PVCs we will fire an alert. - CDIDefaultStorageClassDegraded - when the default SC does not support CSI/Snapshot clone (smart clone) or does not have ReadWriteMany access mode (for live migration). - CDIStorageProfilesIncomplete - add storageClass and provisioner labels. - CDIDataImportCronOutdated - add dataImportCron namespace and name labels. Also: * Rename the metric kubevirt_cdi_storageprofile_status to kubevirt_cdi_storageprofile_info since it always reports value 1, where the label values provide the details about the storage class and storage profile. * Add snapshot manifests for tests and deploy snapshot CRDs in the hpp destructive lane Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

This is a follow-up to kubevirt#2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

* Suppress alerts to reduce noise of dependent ones This is a follow-up to #2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * CR fixes Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Remove expensive func test Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Refactor updateDataImportCronCondition Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Add some verify comments Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Add IsDataVolumeUsingDefaultStorageClass helper for readability Signed-off-by: Arnon Gilboa <agilboa@redhat.com> --------- Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

This is a follow-up to kubevirt#2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

) * Suppress alerts to reduce noise of dependent ones This is a follow-up to #2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * CR fixes Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Remove expensive func test Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Refactor updateDataImportCronCondition Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Add some verify comments Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * Add IsDataVolumeUsingDefaultStorageClass helper for readability Signed-off-by: Arnon Gilboa <agilboa@redhat.com> --------- Signed-off-by: Arnon Gilboa <agilboa@redhat.com> Co-authored-by: Arnon Gilboa <agilboa@redhat.com>

Manual backport of kubevirt#3129 This is a follow-up to kubevirt#2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

) * [release-v1.58] Suppress alerts to reduce noise of dependent ones Manual backport of #3129 This is a follow-up to #2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * [release-v1.58] Get no-provisioner storage capabilities from PVs Manual backport of #3128 It's time to get the storage capabilities for all no-provisioner storage classes based on the existing PVs, and not only for those labeled with `local.storage.openshift.io/owner-name`, as done so far. It will also silence the unnecessary CDIStorageProfilesIncomplete alert for local SC in the CI. Signed-off-by: Arnon Gilboa <agilboa@redhat.com> --------- Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

) * [release-v1.57] Suppress alerts to reduce noise of dependent ones Manual backport of #3129 This is a follow-up to #2998 introducing the following changes to alert rules: - CDIDefaultStorageClassDegraded - do not fire when no default SC (either k8s or virt). - CDIDataImportCronOutdated - do not fire when no default SC (either k8s or virt), as DIC import DVs use default SC. - CDINoDefaultStorageClass - fire not only when there is a pending DV (and PVC) but also when DV has an empty status (waiting for default SC etc.) Signed-off-by: Arnon Gilboa <agilboa@redhat.com> * [release-v1.57] Get no-provisioner storage capabilities from PVs Manual backport of #3128 It's time to get the storage capabilities for all no-provisioner storage classes based on the existing PVs, and not only for those labeled with `local.storage.openshift.io/owner-name`, as done so far. It will also silence the unnecessary CDIStorageProfilesIncomplete alert for local SC in the CI. Signed-off-by: Arnon Gilboa <agilboa@redhat.com> --------- Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Nov 22, 2023

kubevirt-bot requested review from awels and ShellyKa13 November 22, 2023 14:25

kubevirt-bot added the size/XL label Nov 22, 2023

arnongilboa force-pushed the alerts branch from 4d9341b to 0337ca8 Compare November 22, 2023 14:27

sradco reviewed Nov 23, 2023

View reviewed changes

doc/metrics.md Outdated Show resolved Hide resolved

arnongilboa changed the title ~~Add new Prometheus alerts and label existing alerts~~ [WIP] Add new Prometheus alerts and label existing alerts Nov 23, 2023

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 23, 2023

akalenyu reviewed Nov 23, 2023

View reviewed changes

avlitman reviewed Nov 26, 2023

View reviewed changes

akalenyu reviewed Nov 28, 2023

View reviewed changes

arnongilboa force-pushed the alerts branch 2 times, most recently from c69304f to 3a33606 Compare December 4, 2023 12:04

sradco reviewed Dec 5, 2023

View reviewed changes

ShellyKa13 reviewed Dec 7, 2023

View reviewed changes

arnongilboa force-pushed the alerts branch from 3a33606 to 6dfc4cd Compare December 11, 2023 11:36

arnongilboa changed the title ~~[WIP] Add new Prometheus alerts and label existing alerts~~ Add new Prometheus alerts and label existing alerts Dec 11, 2023

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2023

arnongilboa added 2 commits December 12, 2023 11:33

CR fixes

4d200a5

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Create stub VolumeSnapshotClass for testing

b32daa4

Including the VolumeSnapshot/Class/Content crds for the CDIDefaultStorageClassDegraded alert func test. Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

arnongilboa force-pushed the alerts branch from 6dfc4cd to b32daa4 Compare December 12, 2023 09:53

sradco reviewed Dec 12, 2023

View reviewed changes

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 18, 2023

arnongilboa force-pushed the alerts branch from d8d630b to 4cbffc3 Compare December 18, 2023 13:59

arnongilboa added 2 commits December 18, 2023 17:29

CR fixes

d480d88

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

Nicify StorageProfile metric update

4d787e4

Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

arnongilboa force-pushed the alerts branch from 4cbffc3 to 4d787e4 Compare December 18, 2023 15:30

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Dec 19, 2023

kubevirt-bot merged commit edda5ab into kubevirt:main Dec 19, 2023
18 checks passed

arnongilboa mentioned this pull request Dec 25, 2023

[release-v1.58] Add new Prometheus alerts and label existing alerts #3040

Merged

arnongilboa mentioned this pull request Dec 25, 2023

[release-v1.57] Add new Prometheus alerts and label existing alerts #3041

Merged

arnongilboa mentioned this pull request Mar 12, 2024

Suppress alerts to reduce noise of dependent ones #3129

Merged

		metrics.Registry.MustRegister(controller.DefaultVirtStorageClassesGauge)
		metrics.Registry.MustRegister(controller.DefaultStorageClassWithSmartCloneAndRWXGauge)

Add new Prometheus alerts and label existing alerts #2998

Add new Prometheus alerts and label existing alerts #2998

Conversation

arnongilboa commented Nov 22, 2023 • edited Loading

akalenyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avlitman commented Nov 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnongilboa commented Nov 26, 2023

Choose a reason for hiding this comment

arnongilboa Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnongilboa commented Dec 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShellyKa13 Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

arnongilboa Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

sradco Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

kubevirt-bot commented Dec 18, 2023

arnongilboa commented Dec 18, 2023

ShellyKa13 commented Dec 19, 2023

sradco commented Dec 19, 2023

arnongilboa commented Nov 22, 2023 •

edited

Loading

avlitman commented Nov 26, 2023 •

edited

Loading

arnongilboa Dec 4, 2023 •

edited

Loading

ShellyKa13 Dec 11, 2023 •

edited

Loading

arnongilboa Dec 11, 2023 •

edited

Loading

sradco Dec 12, 2023 •

edited

Loading