Fix alert bug #1953

nunnatsa · 2022-05-17T11:45:14Z

Fix Prometheus alert bug, where the KubevirtHyperconvergedClusterOperatorInstallationNotCompletedAlert alert is not working when the HyperConverged CR is not present.

This alert is all about the existence of the HyperConverged CR, and so when it removed with the CR, the alert is not working. The root cause is that HCO manages the PrometheusRule resource as any other resource, and set its owner resource to the HyperConverged CR. In addition, the code that reconciles the PrometheusRule resource is only run when the HypeConverged CR is found.

This PR moves the PrometheusRule handling code to a new package name alerts. Then the new PrometheusRule reconcile function is called regardless the existence of the HyperConverged CR. Also, the PrometheusRule ownerReference is now set to the hco-operator Deployment, so it will be garbage collected on the operator uninstallation.

Reviewer Checklist

Reviewers are supposed to review the PR for every aspect below one by one. To check an item means the PR is either "OK" or "Not Applicable" in terms of that item. All items are supposed to be checked before merging a PR.

Release note:

Fix Prometheus alert bug, where the KubevirtHyperconvergedClusterOperatorInstallationNotCompletedAlert alert is not working when the HyperConverged CR is not present.

coveralls · 2022-05-17T11:51:11Z

Pull Request Test Coverage Report for Build 2370633377

265 of 364 (72.8%) changed or added relevant lines in 7 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.5%) to 83.781%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/util/cluster.go	1	10	10.0%
controllers/alerts/alerts.go	169	180	93.89%
controllers/hyperconverged/hyperconverged_controller.go	3	14	21.43%
pkg/util/util.go	0	26	0.0%
pkg/util/own_resources.go	85	127	66.93%

Files with Coverage Reduction	New Missed Lines	%
controllers/operands/operandHandler.go	1	88.1%

Totals
Change from base Build 2369603650:	-0.5%
Covered Lines:	4246
Relevant Lines:	5068

💛 - Coveralls

hco-bot · 2022-05-17T15:14:16Z

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure
hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

kubevirt-bot · 2022-05-17T15:14:21Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-gcp, ci/prow/hco-e2e-image-index-sno-azure, ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure
hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-05-18T10:17:10Z

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

kubevirt-bot · 2022-05-18T10:17:13Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-05-18T10:38:15Z

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure
hco-e2e-image-index-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-gcp

kubevirt-bot · 2022-05-18T10:38:19Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-gcp, ci/prow/hco-e2e-image-index-sno-azure, ci/prow/hco-e2e-upgrade-index-sno-azure

In response to this:

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure
hco-e2e-image-index-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2022-05-18T17:43:52Z

/retest

hco-bot · 2022-05-18T18:56:52Z

hco-e2e-image-index-gcp lane succeeded.
/override ci/prow/hco-e2e-image-index-aws

kubevirt-bot · 2022-05-18T18:57:01Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-aws

In response to this:

hco-e2e-image-index-gcp lane succeeded.
/override ci/prow/hco-e2e-image-index-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2022-05-18T18:58:37Z

/retest

hco-bot · 2022-05-18T19:13:55Z

hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

kubevirt-bot · 2022-05-18T19:14:00Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-sno-azure

In response to this:

hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

controllers/alerts/alerts.go

controllers/hyperconverged/hyperconverged_controller_test.go

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

Add the new OwnResources object within the ClusterInfo to provide the running Pod, its Deployment and the CSV that owns the Deployment. Added tree new methods to the ClusterInfo interface: * GetPod() *corev1.Pod * GetDeployment() *appsv1.Deployment * GetCSV() *csvv1alpha1.ClusterServiceVersion Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

Move the handling of the the Prometheus Rule to a a new package and call it from the main reconcile function before reading the HyperConverged CR. Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

sonarcloud · 2022-05-23T10:30:36Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

hco-bot · 2022-05-23T11:54:06Z

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

nunnatsa · 2022-05-23T11:54:07Z

/test okd-hco-e2e-image-index-aws
/test okd-hco-e2e-image-index-gcp

kubevirt-bot · 2022-05-23T11:54:09Z

@nunnatsa: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build-hco-test-utils-image
/test pull-hyperconverged-cluster-operator-e2e-k8s-1.23

Use /test all to run the following jobs that were automatically triggered:

pull-hyperconverged-cluster-operator-e2e-k8s-1.23

In response to this:

/test okd-hco-e2e-image-index-aws
/test okd-hco-e2e-image-index-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot · 2022-05-23T11:54:09Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-05-23T12:25:15Z

hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

kubevirt-bot · 2022-05-23T12:25:17Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-sno-azure

In response to this:

hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-05-23T13:08:28Z

okd-hco-e2e-image-index-aws lane succeeded.
/override ci/prow/okd-hco-e2e-image-index-gcp

kubevirt-bot · 2022-05-23T13:08:32Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/okd-hco-e2e-image-index-gcp

In response to this:

okd-hco-e2e-image-index-aws lane succeeded.
/override ci/prow/okd-hco-e2e-image-index-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiraboschi · 2022-05-23T13:25:10Z

ignoring slightly decreased coverage
/override coverage/coveralls

kubevirt-bot · 2022-05-23T13:25:13Z

@tiraboschi: Overrode contexts on behalf of tiraboschi: coverage/coveralls

In response to this:

ignoring slightly decreased coverage
/override coverage/coveralls

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-05-23T13:43:04Z

@nunnatsa: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hco-e2e-upgrade-prev-index-azure	`ee16865`	link	true	`/test hco-e2e-upgrade-prev-index-azure`
ci/prow/hco-e2e-image-index-sno-azure	`ee16865`	link	false	`/test hco-e2e-image-index-sno-azure`
ci/prow/okd-hco-e2e-image-index-gcp	`ee16865`	link	true	`/test okd-hco-e2e-image-index-gcp`
ci/prow/hco-e2e-upgrade-index-azure	`ee16865`	link	true	`/test hco-e2e-upgrade-index-azure`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

nunnatsa · 2022-05-23T18:08:46Z

/override-bot

nunnatsa · 2022-05-23T18:12:46Z

/override-bot

nunnatsa · 2022-05-23T18:14:55Z

hco-e2e-upgrade-index-aws lane passed
/override ci/prow/hco-e2e-upgrade-index-azure

kubevirt-bot · 2022-05-23T18:14:58Z

@nunnatsa: Overrode contexts on behalf of nunnatsa: ci/prow/hco-e2e-upgrade-index-azure

In response to this:

hco-e2e-upgrade-index-aws lane passed
/override ci/prow/hco-e2e-upgrade-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiraboschi · 2022-05-24T08:41:18Z

/approve

kubevirt-bot · 2022-05-24T08:42:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tiraboschi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tiraboschi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels May 17, 2022

kubevirt-bot requested review from orenc1 and tiraboschi May 17, 2022 11:45

kubevirt-bot added the size/XXL label May 17, 2022

nunnatsa force-pushed the fix-alert-bug branch from f5dcb5c to 7289025 Compare May 17, 2022 11:50

nunnatsa changed the title ~~Fix alert bug~~ WIP Fix alert bug May 17, 2022

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 17, 2022

nunnatsa force-pushed the fix-alert-bug branch 2 times, most recently from 2763dac to bddb208 Compare May 17, 2022 13:35

nunnatsa force-pushed the fix-alert-bug branch from bddb208 to 94a83a1 Compare May 18, 2022 08:53

nunnatsa changed the title ~~WIP Fix alert bug~~ Fix alert bug May 18, 2022

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 18, 2022

nunnatsa force-pushed the fix-alert-bug branch from 94a83a1 to bbf72e9 Compare May 18, 2022 17:23

tiraboschi reviewed May 19, 2022

View reviewed changes

controllers/alerts/alerts.go Show resolved Hide resolved

controllers/hyperconverged/hyperconverged_controller_test.go Show resolved Hide resolved

nunnatsa added 2 commits May 23, 2022 07:23

add new Makefile target to run goimport

5a04f41

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

nunnatsa and others added 3 commits May 23, 2022 07:23

Reconcile the Prometheus Rules regrdless the present of the HC CR

c5a29e0

Move the handling of the the Prometheus Rule to a a new package and call it from the main reconcile function before reading the HyperConverged CR. Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

add events and unit tests

d158ff3

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

apply code review comments

ee16865

Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

nunnatsa force-pushed the fix-alert-bug branch from bbf72e9 to ee16865 Compare May 23, 2022 10:29

tiraboschi approved these changes May 24, 2022

View reviewed changes

kubevirt-bot assigned tiraboschi May 24, 2022

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label May 24, 2022

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 24, 2022

kubevirt-bot merged commit b0f92d2 into kubevirt:main May 24, 2022

nunnatsa deleted the fix-alert-bug branch May 24, 2022 10:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix alert bug #1953

Fix alert bug #1953

nunnatsa commented May 17, 2022 •

edited

Loading

coveralls commented May 17, 2022 •

edited

Loading

hco-bot commented May 17, 2022

kubevirt-bot commented May 17, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

nunnatsa commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

nunnatsa commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

sonarcloud bot commented May 23, 2022

hco-bot commented May 23, 2022

nunnatsa commented May 23, 2022

kubevirt-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

hco-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

hco-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

tiraboschi commented May 23, 2022

kubevirt-bot commented May 23, 2022

openshift-ci bot commented May 23, 2022

nunnatsa commented May 23, 2022

nunnatsa commented May 23, 2022

nunnatsa commented May 23, 2022

kubevirt-bot commented May 23, 2022

tiraboschi commented May 24, 2022

kubevirt-bot commented May 24, 2022

Fix alert bug #1953

Fix alert bug #1953

Conversation

nunnatsa commented May 17, 2022 • edited Loading

coveralls commented May 17, 2022 • edited Loading

Pull Request Test Coverage Report for Build 2370633377

💛 - Coveralls

hco-bot commented May 17, 2022

kubevirt-bot commented May 17, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

nunnatsa commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

nunnatsa commented May 18, 2022

hco-bot commented May 18, 2022

kubevirt-bot commented May 18, 2022

sonarcloud bot commented May 23, 2022

hco-bot commented May 23, 2022

nunnatsa commented May 23, 2022

kubevirt-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

hco-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

hco-bot commented May 23, 2022

kubevirt-bot commented May 23, 2022

tiraboschi commented May 23, 2022

kubevirt-bot commented May 23, 2022

openshift-ci bot commented May 23, 2022

nunnatsa commented May 23, 2022

nunnatsa commented May 23, 2022

nunnatsa commented May 23, 2022

kubevirt-bot commented May 23, 2022

tiraboschi commented May 24, 2022

kubevirt-bot commented May 24, 2022

nunnatsa commented May 17, 2022 •

edited

Loading

coveralls commented May 17, 2022 •

edited

Loading