Expose HCO monitoring on plain k8s

[assafad]

What this PR does / why we need it:
Right now we are creating HCO monitoring-related resources only in OpenShift clusters, and thus HCO metrics and alerts are created and exposed to Prometheus only on these clusters. On plain k8s clusters, HCO only implements metrics, but they are not exposed to Prometheus, and alerts are not created or reconciled at all.
This PR exposes metrics and alerts also on k8s clusters, by enabling the creation and reconciliation of required monitoring-related resources (e.g. PrometheusRule,ServiceMonitor), as long as Prometheus is installed on the cluster.

Reviewer Checklist

Reviewers are supposed to review the PR for every aspect below one by one. To check an item means the PR is either "OK" or "Not Applicable" in terms of that item. All items are supposed to be checked before merging a PR.

• PR Message
• Commit Messages
• How to test
• Unit Tests
• Functional Tests
• User Documentation
• Developer Documentation
• Upgrade Scenario
• Uninstallation Scenario
• Backward Compatibility
• Troubleshooting Friendly

Jira Ticket: https://issues.redhat.com/browse/CNV-26009

Release note:

Expose HCO monitoring on plain k8s

[assafad]

/hold

[assafad]

Tested with kubevirtci and kind clusters. Steps to view metrics and alerts using kubevirtci:

• export KUBEVIRT_DEPLOY_PROMETHEUS=true
• make cluster-up
• make cluster-sync
• in order to expose alerts to Prometheus, edit Prometheus to select all existing prometheusrules:
  • ./cluster/kubectl.sh edit prometheus k8s -n monitoring
  • edit

  ruleSelector:
     matchLabels:
      	prometheus: k8s
      	role: alert-rules
  

  to ruleSelector: {}
• ./cluster/kubectl.sh port-forward service/prometheus-k8s 9090:9090 -n monitoring
• access Prometheus dashboard using http://localhost:9090

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[coveralls]

Pull Request Test Coverage Report for Build 5337253780

• 23 of 34 (67.65%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.06%) to 85.903%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
controllers/alerts/rbac.go	8	9	88.89%
pkg/util/cluster.go	14	17	82.35%
controllers/hyperconverged/hyperconverged_controller.go	0	7	0.0%

Totals
Change from base Build 5330400492:	-0.06%
Covered Lines:	4942
Relevant Lines:	5753

---

💛 - Coveralls

[assafad]

Hi @nunnatsa, @machadovilaca, Could you please review this PR?

[assafad]

/retest

[nunnatsa]

/retest

[nunnatsa]

Great job!
/lgtm

[openshift-ci]

@assafad: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hco-e2e-kv-smoke-azure	5665934	link	true	/test hco-e2e-kv-smoke-azure

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[nunnatsa]

/override coverage/coveralls
/approve

[kubevirt-bot]

@nunnatsa: Overrode contexts on behalf of nunnatsa: coverage/coveralls

In response to this:

/override coverage/coveralls
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nunnatsa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nunnatsa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nunnatsa]

hco-e2e-kv-smoke-gcp lane passed

/override ci/prow/hco-e2e-kv-smoke-azure

[kubevirt-bot]

@nunnatsa: Overrode contexts on behalf of nunnatsa: ci/prow/hco-e2e-kv-smoke-azure

In response to this:

hco-e2e-kv-smoke-gcp lane passed

/override ci/prow/hco-e2e-kv-smoke-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[assafad]

/unhold

[kubevirt-bot]

@assafad: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-hyperconverged-cluster-operator-e2e-k8s-1.26-centos9	5665934	link	unknown	/test pull-hyperconverged-cluster-operator-e2e-k8s-1.26-centos9
pull-hyperconverged-cluster-operator-e2e-k8s-1.27	5665934	link	unknown	/test pull-hyperconverged-cluster-operator-e2e-k8s-1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.