Add multi-namespace cache support

[charith-elastic]

Uses the multi-namespace cache if the operator is provided with a list of namespaces to be managed. This would allow users to restrict the operator to a small set of namespaces.

For backward compatibility reasons, namespace flag has been retained but it will be ignored if the namespace-list flag is provided.

Also changes the E2E test runner to have a single operator for all E2E namespaces rather than an operator per namespace.

Fixes #405, #782

[anyasabo]

I know in the past we discussed maybe wanting two separate flags since we chose the singular namespace. I think because we are still in beta it would be worthwhile to have the breaking change and only support namespaces. That way we do not need to have the potentially confusing mix of namespace and namespaces with one taking priority over the other, and we also have less code. I'd also rather do breaking changes sooner than later so we have more time to iron out any issues. What do you think?

[anyasabo]

I'm not too familiar with how E2EImage was populated. Were we reusing image names across test runs and so they were getting cached even if they changed?

[charith-elastic]

Image caching is a problem if you manually run the tests with uncommitted changes.

I kept the setting because it won't affect the normal CI runs (E2E image name is unique and will require a pull from the registry anyway).

[sebgl]

I think we may have introduced IfNotPresent a while ago in order to run tests on Minikube. We didn't have a registry there, so we used to build the image in Minikube context. Meaning it was possible to use it, but not to pull it from Minikube.
The situation is different now, we do run a registry in Minikube if I remember correctly (hack/registry.sh).

[charith-elastic]

I agree with @anyasabo about not having both namespace and namespace-list in the long term. My thinking is that we should keep both flags until the GA release is confirmed, and then drop the namespace flag and (maybe) rename namespace-list to namespaces.

[sebgl]

I think we may have introduced IfNotPresent a while ago in order to run tests on Minikube. We didn't have a registry there, so we used to build the image in Minikube context. Meaning it was possible to use it, but not to pull it from Minikube.
The situation is different now, we do run a registry in Minikube if I remember correctly (hack/registry.sh).

[sebgl]

Regarding breaking changes: I think I would also vote for removing the singular namespace right now.

Arguments:

• Most users will probably upgrade to next release by applying the newer operator yaml resource, so they won't notice.
• If they apply the new operator manifest with a singular namespace, I think the error will be explicit in the operator pod logs.
• Doing it in 2-phases as @charith-elastic suggested sounds like the right way to deal with api changes in general. In this particular case I feel like it's not strictly required though (it's not "really" the public user API), and will actually be a breaking change in 2 releases (post-GA, once we remove the deprecated flag). Might as well break it now IMO.

[thbkrkr]

LGTM

I agree with the small breakage since we are the ones who ship the operator yaml.