Disable caching of Secret and ConfigMap resources

[hiddeco]

At present all controllers making use of Secret and/or ConfigMap references cache all these resources. Depending on the cluster, this may result in excessive memory usage while only a portion of the resources is actually being used. Especially when the cluster has for example a high number of Helm releases.

To reduce the memory footprint, we have agreed to disable the caching of these resources with an optional opt-in to enable it for clusters where this overhead is acceptable. The opt-in should rely on the same feature flag across controllers (even if they do not make use of both resource kinds), which is CacheSecretsAndConfigMaps.

• source-controller: Disable caching of secrets and configmaps source-controller#989
• helm-controller: Disable caching of secrets and configmaps helm-controller#513
• kustomize-controller: Disable caching of Secrets and ConfigMaps kustomize-controller#789
• notification-controller: Disable caching of Secrets and ConfigMaps notification-controller#468
• image-automation-controller: Disable caching of Secrets and ConfigMaps image-automation-controller#479
• image-reflector-controller: Disable caching of Secrets and ConfigMaps image-reflector-controller#337

[hiddeco]

Completed, further release is tracked in #3533.