NETOBSERV-565 added service monitors for flp prometheus and console plugin

[KalmanMeth]

To view the metrics in the console, apply the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

[jpinsonneau]

Suggested change

	Namespace: constants.DefaultOperatorNamespace,
	Namespace: b.namespace,

Namespace should be taken from CRD

[jpinsonneau]

LGTM in terms of code, not tested yet. Thanks @KalmanMeth !

[jotak]

I'm going to test it
thanks for the PR @KalmanMeth

[jotak]

@KalmanMeth it doesn't work for me, there is a permission issue when trying to read the servicemonitor CRD to determine if they can be installed.
In fact, we use a discovery mechanism to check when external APIs are available (we already do this to check whether ConsolePlugin API exists, for instance) : look at this file: https://github.com/netobserv/network-observability-operator/blob/main/pkg/discover/apis.go

So to check ServiceMonitor, you can edit this file to add HasServiceMonitor, similar to the existing HasConsole / HasCNO. Doing this way, we don't have to add a new read permission on CRD kind itself.

[jotak]

There's something more to do: manage namespace changes. It is possible to update the netobserv installation to change its namespace, so what the operator does in that case is to remove all the created resources and recreate them in the other namespace.

You don't have to implement all of this, you just need to declare the servicemonitor resource when the reconcilers are created, see here: https://github.com/netobserv/network-observability-operator/blob/main/controllers/consoleplugin/consoleplugin_reconciler.go#L32-L56

• you need to add a reference in ownedObjects
• you need to call nobjMngr.AddManagedObject

once it's done, upon namespace change, the controller will take care of deleting these resources and re-creating them.

Same thing to do for FLP monolith, FLP ingester and FLP transformer

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jotak by writing /assign @jotak in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[KalmanMeth]

@KalmanMeth it doesn't work for me, there is a permission issue when trying to read the servicemonitor CRD to determine if they can be installed. In fact, we use a discovery mechanism to check when external APIs are available (we already do this to check whether ConsolePlugin API exists, for instance) : look at this file: https://github.com/netobserv/network-observability-operator/blob/main/pkg/discover/apis.go

So to check ServiceMonitor, you can edit this file to add HasServiceMonitor, similar to the existing HasConsole / HasCNO. Doing this way, we don't have to add a new read permission on CRD kind itself.

@jotak I added code to check for HasServiceMonitor. I'm not sure what the environment was in which you got a failure, so I could not check it.

[jotak]

@KalmanMeth sorry for the delay, I'll take another look today

[jotak]

So I still get the same error (plus also an error saying missing permissions to get/create servicemonitors)
I get this error with the nominal test deployment, which is deploying the operator on a running cluster + the default FlowCollector resource. Basically I just build & push the image, then run IMG=<that image> make deploy deploy-sample-cr

The exact error I get (in logs) is:

W1122 08:20:08.309526       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:netobserv:netobserv-controller-manager" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E1122 08:20:08.309548       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.CustomResourceDefinition: failed to list *v1.CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:netobserv:netobserv-controller-manager" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

This is logical: we must tell that we need extra permissions, to read/write ServiceMonitors. This is done via kubebuilder annotations, cf in flowcollector_controller.go L58-69 : you need to add a new line

//+kubebuilder:rbac:groups=monitoring.coreos.com,resources=servicemonitors,verbs=get;create;delete;update;patch;list

Then run:

make generate

It will amend automatically config/rbac/role.yaml to add new permissions. This will then be used when deploying the operator to a running cluster.

The other thing is, my previous comment to add the discovery mechanism was to avoid having to read the servicemonitor CRD as you do in consoleplugin_reconciler.go (L202-210) and flp_service_monitor.go (L29-37). So we should remove these lines, as it saves us from requiring an extra permission to read customresourcedefinitions.apiextensions.k8s.io.

[eranra]

@KalmanMeth this will not be needed once we merge #203

[KalmanMeth]

So I still get the same error (plus also an error saying missing permissions to get/create servicemonitors)
I get this error with the nominal test deployment, which is deploying the operator on a running cluster + the default FlowCollector resource. Basically I just build & push the image, then run IMG=<that image> make deploy deploy-sample-cr

@jotak I made updates according to your comments. I hope it is now satisfactory.

[jotak]

Hi Kalman,
Thanks for the update, it now deploys correctly without an error. I get the servicemonitors created and FLP metrics feeding prometheus 👍
But I see 2 issues:

1. I don't get the console plugin metrics. It seems to be a certificate issue. Not sure what exactly the problem is.. Setting InsecureSkipVerify: true would work, but that's not a good fix.
2. When I deploy with Kafka, we loose the FLP metrics. At first glance it seems because the FLP ServiceMonitor is not correctly adapted for this case (when using Kafka, the FLP deployment model changes from monolithic to publisher/subscriber, that's the reason why we have this distinction of flp-ingester / flp-transformer in code, and we need to adapt the service monitor to point to these workloads.

Let me know if you have an idea for point 1. For point 2, I can try doing a fix and adding it to your PR (I know this part of the code is tricky to apprehend) ; If we can't get that fixed quickly we can go ahead & merge, and address them in follow-ups

[jotak]

/lgtm
thanks!

Follow-up JIRA for the console plugin issue: https://issues.redhat.com/browse/NETOBSERV-765