Enable operator metrics collection #273

OlivierCazade · 2023-02-17T13:14:53Z

Enable collection for operator metrics.

jotak · 2023-02-17T17:55:47Z

config/prometheus/monitor.yaml

@@ -12,7 +12,6 @@ spec:
    - path: /metrics
      port: https
      scheme: https


What is missing for having a TLS without skipping verify?
It's basically the same reason why we have this bug: https://issues.redhat.com/browse/NETOBSERV-765

Similar to what we did for FLP we need to add the labels to the metric services to generate certificate and mount them in the kube-rbac-proxy container so they can be used.

OlivierCazade · 2023-03-09T16:18:48Z

Added SSL and auth configuration for downstream collection, this made operator metrics endpoint incompatible with monitoring user workload.

@memodi appart from review comments, the PR is ready to be tested.
After deploying this branch and setting DOWNSTREAM_DEPLOYMENT to true, you should see operator metrics without enabling user workload.

codecov · 2023-03-09T16:20:59Z

Codecov Report

Merging #273 (434ced9) into main (8a4db3c) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #273      +/-   ##
==========================================
- Coverage   47.61%   47.57%   -0.04%     
==========================================
  Files          43       43              
  Lines        5015     5019       +4     
==========================================
  Hits         2388     2388              
- Misses       2416     2420       +4     
  Partials      211      211

Flag	Coverage Δ
unittests	`47.57% <0.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
controllers/flowcollector_controller.go	`52.73% <0.00%> (ø)`
controllers/flowcollector_objects.go	`17.77% <0.00%> (-1.74%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

memodi · 2023-03-09T16:41:31Z

Added SSL and auth configuration for downstream collection, this made operator metrics endpoint incompatible with monitoring user workload.

@OlivierCazade - is the idea, they would be compatible with SSL and auth configuration once operator is deployed with DOWNSTREAM_DEPLOYMENT to true?

OlivierCazade · 2023-03-09T17:29:08Z

Yes, SSL and authentication for the metrics endpoint. And the prometheus deployment should automatically use it from the ServiceMonitor configuration

memodi · 2023-03-09T17:41:48Z

/ok-to-test

github-actions · 2023-03-09T17:44:16Z

New image: ["quay.io/netobserv/network-observability-operator:1b5e2fb"]. It will expire after two weeks.

memodi · 2023-03-09T18:04:33Z

@OlivierCazade - I tried with image from this PR, I am not able to see those metrics:

$ oc get pod/netobserv-controller-manager-7794cf659b-vcnhn -o yaml | egrep -A 1 "DOWN"
    - --downstream-deployment=$(DOWNSTREAM_DEPLOYMENT)
    command:
--
    - name: DOWNSTREAM_DEPLOYMENT
      value: "true"

$ oc get servicemonitor
NAME                        AGE
netobserv-metrics-monitor   5m19s

am I missing something?

OlivierCazade · 2023-03-10T11:43:16Z

Most of the changes in this PR are at the bundle level, did you also used the bundle from this branch?

memodi · 2023-03-13T14:27:59Z

/ok-to-test

github-actions · 2023-03-13T14:32:58Z

New images:

quay.io/netobserv/network-observability-operator:a5309c1
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-a5309c1
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-a5309c1

They will expire after two weeks.

Catalog source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-a5309c1
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2023-03-13T15:31:49Z

/ok-to-test

github-actions · 2023-03-13T15:36:56Z

New images:

quay.io/netobserv/network-observability-operator:7a7c5bf
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-7a7c5bf
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-7a7c5bf

They will expire after two weeks.

Catalog source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-7a7c5bf
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2023-03-13T16:31:03Z

/label qe-approved

Verified NOO metrics and FLP metrics are getting scrapped by cluster prometheus when DOWNSTREAM_DEPLOYMENT == true

jotak · 2023-03-14T08:41:04Z

monitor.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1


I don't think we want this file, do we? Is it included in the bundles, or is it just there as a helper (e.g. for make targets) ? afaict it should not be included in bundles

No sorry, this was one of my test file.

jotak · 2023-03-14T08:44:15Z

config/openshift-olm/monitor.yaml

      tlsConfig:
-        insecureSkipVerify: true
+        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
+        serverName: netobserv-metrics-service.netobserv.svc


so, my understanding is that we should hardcode here the installation namespace, openshift-operators, right?

OlivierCazade · 2023-03-14T17:15:51Z

/approve

openshift-ci · 2023-03-14T17:16:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: OlivierCazade

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [OlivierCazade]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

memodi · 2023-03-22T14:59:08Z

adding NETOBSERV-900 as JIRA reference

OlivierCazade requested review from jotak and jpinsonneau February 17, 2023 13:14

OlivierCazade force-pushed the operator-metrics branch from 39d7338 to 55bef5b Compare February 17, 2023 13:38

jotak reviewed Feb 17, 2023

View reviewed changes

memodi mentioned this pull request Mar 3, 2023

NETOBSERV-736: Added downstream deployment configuration for monitoring collection #282

Merged

OlivierCazade force-pushed the operator-metrics branch from 55bef5b to 5db2a1c Compare March 9, 2023 16:15

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 9, 2023

OlivierCazade force-pushed the operator-metrics branch from 5db2a1c to fcc89e9 Compare March 13, 2023 11:55

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 13, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 13, 2023

OlivierCazade force-pushed the operator-metrics branch from fcc89e9 to 185b7a6 Compare March 13, 2023 15:30

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 13, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 13, 2023

openshift-ci bot added the qe-approved QE has approved this pull request label Mar 13, 2023

jotak reviewed Mar 14, 2023

View reviewed changes

jotak mentioned this pull request Mar 14, 2023

Allow modifying namespace from web console #281

Closed

OlivierCazade force-pushed the operator-metrics branch from 185b7a6 to 21cb6cd Compare March 14, 2023 15:46

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Mar 14, 2023

OlivierCazade added 2 commits March 14, 2023 17:01

Enable operator metrics collection

4cf4031

Added operator metrics ssl and auth configuration

434ced9

OlivierCazade force-pushed the operator-metrics branch from 21cb6cd to 434ced9 Compare March 14, 2023 16:07

jotak approved these changes Mar 14, 2023

View reviewed changes

openshift-ci bot assigned jotak Mar 14, 2023

openshift-ci bot added the lgtm label Mar 14, 2023

openshift-ci bot added the approved label Mar 14, 2023

openshift-merge-robot merged commit 4e15d42 into netobserv:main Mar 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable operator metrics collection #273

Enable operator metrics collection #273

OlivierCazade commented Feb 17, 2023

jotak Feb 17, 2023

OlivierCazade Feb 20, 2023

OlivierCazade commented Mar 9, 2023

codecov bot commented Mar 9, 2023 •

edited

Loading

memodi commented Mar 9, 2023

OlivierCazade commented Mar 9, 2023

memodi commented Mar 9, 2023

github-actions bot commented Mar 9, 2023

memodi commented Mar 9, 2023

OlivierCazade commented Mar 10, 2023

memodi commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

memodi commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

memodi commented Mar 13, 2023 •

edited

Loading

jotak Mar 14, 2023

OlivierCazade Mar 14, 2023

jotak Mar 14, 2023

OlivierCazade commented Mar 14, 2023

openshift-ci bot commented Mar 14, 2023

memodi commented Mar 22, 2023 •

edited by openshift-ci bot

Loading

Enable operator metrics collection #273

Enable operator metrics collection #273

Conversation

OlivierCazade commented Feb 17, 2023

jotak Feb 17, 2023

Choose a reason for hiding this comment

OlivierCazade Feb 20, 2023

Choose a reason for hiding this comment

OlivierCazade commented Mar 9, 2023

codecov bot commented Mar 9, 2023 • edited Loading

Codecov Report

memodi commented Mar 9, 2023

OlivierCazade commented Mar 9, 2023

memodi commented Mar 9, 2023

github-actions bot commented Mar 9, 2023

memodi commented Mar 9, 2023

OlivierCazade commented Mar 10, 2023

memodi commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

memodi commented Mar 13, 2023

github-actions bot commented Mar 13, 2023

memodi commented Mar 13, 2023 • edited Loading

jotak Mar 14, 2023

Choose a reason for hiding this comment

OlivierCazade Mar 14, 2023

Choose a reason for hiding this comment

jotak Mar 14, 2023

Choose a reason for hiding this comment

OlivierCazade commented Mar 14, 2023

openshift-ci bot commented Mar 14, 2023

memodi commented Mar 22, 2023 • edited by openshift-ci bot Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

memodi commented Mar 13, 2023 •

edited

Loading

memodi commented Mar 22, 2023 •

edited by openshift-ci bot

Loading