Helm: Use informer to list helm secrets to improve performance

[Lucaber]

Description of the change:

Helm stores its state in secrets inside the cluster.
Instead of listing these secrets before every reconciliation of every release, we use an informer to query a local secrets list.

Whats the current status of the helm-operator-plugins repo, should I also create a MR there?

Motivation for the change:

We are running 2 helm-operators with 50 CRs each in a namespace with over 1000 secrets.
Listing these secrets, even with a filter, takes more than 1 second.
Running multiple of these list queries in parallel generates a high load on the kubernetes apiserver and etcd.
Combined with the default reconcile period of 1m this results in a constant high load.

Before and After metrics (after installing this MR on the first and second helm-operator in the cluster):

This issue was primarily noticed due to a very high load and network traffic between the kubernetes etcd instances.

Checklist

If the pull request includes user-facing changes, extra documentation is required:

• Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
• Add or update relevant sections of the docs website in website/content/en/docs

[Lucaber]

In helm-operator-plugins, we already have an informer for secrets because we watch secrets in the controller.

Could you point me to the file where the watch for Secrets in helm-operator-plugins is setup?

I wonder if we should watch secrets here as well. If we did that, we would need a SecretsInterface implementation that wraps mgr.GetClient() so that we reuse the underlying/existing informer and cache.

What do you think about that approach?

Yes, wrapping the SecretsInterface seams to be best option, as we don't need to copying a lot of internal code from helm itself to for example write a custom driver.

[Lucaber]

Any new updates? Somehow @joelanford was automatically removed as a reviewer when "re-requesting" the review.

[everettraven]

@Lucaber Thanks for the contribution! Apologies for this PR getting lost in the weeds of other PRs. I'll try to get this pushed through.

/lgtm