Bug creating metric service monitors when watching a different namespace

[davidkirwan]

Bug Report

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).
I ran an operator from namespace A, which is set to watch namespace B.

What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).
I expected to see a ServiceMonitor to be created on the operator metrics service in namespace A

What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).
I saw an error in the operator logs about it being unable to create the servicemonitor:

{"level":"info","ts":1580740974.4433243,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"the namespace of the provided object does not match the namespace sent on the request"}

Environment

• operator-sdk version:

operator-sdk version: "v0.12.0", commit: "2445fcda834ca4b7cf0d6c38fba6317fb219b469", go version: "go1.12.13 linux/amd64"

• go version:

go version go1.13.6 linux/amd64

• Kubernetes version information:

kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+b294fe5", GitCommit:"b294fe5", GitTreeState:"clean", BuildDate:"2020-01-06T18:46:07Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

oc version
Client Version: openshift-clients-4.2.2-201910250432-12-g72076900
Server Version: 4.2.14
Kubernetes Version: v1.14.6+b294fe5

• Kubernetes cluster kind:
  Openshift: 4.2.14
  Kubernetes: v1.14.6+b294fe5

• Are you writing your operator in ansible, helm, or go?
  I'm writing the operator in Go

Possible Solution

The solution which I have used as a workaround in my operator, is to pass the namespace that hte operator is running inside, to the function which creates the servicemonitor in the main.go. See PR:

• Use the namespace the operator is running from when creating servicem… #2492
• https://github.com/integr8ly/cloud-resource-operator/pull/103/files#diff-9e63f7b0791edcb4a53c944552b88970R147

Additional context
Add any other context about the problem here.

I've attached some log showing the steps to replicate:
test-operator-64b7c777cc-prqz9-test-operator.log

[camilamacedo86]

I will check it as soon as possible @davidkirwan.
Thank you for your contribution.

[camilamacedo86]

Hi @davidkirwan,

As we spoke about it, following the explanation to attend the need that you are looking for which is; have a cluster-scope operator which will watch/cache just a few namespaces in the cluster and not all.

Note that the operator will still with a cluster-scoped and then, is required to following the steps in the doc. After that, to define(restrict) the few namespaces that you would like to be watching/caching in the project you can use the manager.Options MultiNamespacedCacheBuilder. So, in the main.go you can get the value from a new ENV VAR for example added in the operator.yaml with the list of the namespaces that would be watching/caching to pass to the manager. Following an example.

// List of Namespaces 
// Here you can add all namespaces that the Operator will be dealing with
// Note that the namespace where the operator is deployed should be part of this context as well. 
var namespaces []string 

// Create a new Cmd to provide shared dependencies and start components
mgr, err := manager.New(cfg, manager.Options{
   NewCache: cache.MultiNamespacedCacheBuilder(namespaces),
   MapperProvider:     restmapper.NewDynamicRESTMapper,
   MetricsBindAddress: fmt.Sprintf("%s:%d", metricsHost, metricsPort),
})

This list must contain the operator namespace as well. See here that the metrics resources which are created by default will be done for the operator and its CR's which means that it needs to be able to watch/cache the namespace where it is deployed too.

In this way, I am closing this one as sorted out. Please, feel free to re-open or ping us to do that if you see that still required any help with.

[rcernich]

Sorry to glob onto this, but I'm having a similar issue and am getting the following error (v0.15.2):

{"level":"info","ts":1582742893.4511955,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"an empty namespace may not be set during creation"}

The explanation listed above doesn't seem applicable, as our operator is watching all namespaces (hence, namespace=""). Furthermore, even if watching multiple namespaces, the namespace variable is still set to an empty string.

I can workaround the issue by specifying the operator namespace, but I don't know enough about the service monitors to know what that might break.

Please advise.

[camilamacedo86]

Hi @rcernich,

Your scenario is not the same as this issue, However, your scenario is exactly the same of #2602.

Please, feel free to follow up and contribute with.

[rcernich]

@camilamacedo86, thanks. That's exactly the issue I'm having. I simply changed the code to use the operator namespace when creating the service monitors.