Ramen controllers are caching secret/configmaps from all namespaces

[akalenyu]

Initally sparked by a casual conversation with @nirs about a bug around secret objects,
I took a peek at the source code and it indeed looks like some controllers are caching all configmaps and secrets.
This both:

• Aggressively increases memory usage (think: production cluster of an org with infinite CMs/Secrets)
• Poses a security threat with Ramen having access to all user sensitive data in the cluster - rbac practice reference

This can be solved by instead only caching the secrets/configmaps in the ramen namespace
From a quick read, ramen only cares about those anyway

ramen/controllers/drcluster_controller.go

Lines 162 to 165 in da2b47b

	func (r *DRClusterReconciler) drClusterSecretMapFunc(ctx context.Context, obj client.Object) []reconcile.Request {
	if obj.GetNamespace() != RamenOperatorNamespace() {
	return []reconcile.Request{}
	}

This can be demonstrated in one of the default drenv environments (I used test/envs/regional-dr-kubevirt.yaml):

for i in {1..200}; do kubectl --context dr1 create cm test-cm-$i -n default --from-file=../manifests/largedatafile.txt ; done

$ kubectl get pods --context dr1 -n ramen-system -w
ramen-dr-cluster-operator-896d8c9f6-krbtd   2/2     Running             0             11s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     OOMKilled           0             34s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     Running             1 (1s ago)    35s
ramen-dr-cluster-operator-896d8c9f6-krbtd   2/2     Running             1 (7s ago)    41s
ramen-dr-cluster-operator-896d8c9f6-krbtd   1/2     OOMKilled           1 (31s ago)   65s

May have to kill the existing ramen pod.
Could also watch the memory usage grow with minikube addons enable metrics-server --profile dr1
and then kubectl --context dr1 top pod -n ramen-system

See also

• https://bugzilla.redhat.com/2283489 Bug causing creation of up to 47,000 secrets consuming 329 MiB, leading to OOM in ramen and other operators.

[akalenyu]

/assign akalenyu