helm: fix a memory leak resulting from too many k8s client instantiations

[misberner]

See operator-framework/helm-operator-plugins#198 for a detailed description of the issue. This commit ports over the relevant changes.

I assume that eventually, the generic helm-operator code in this repo should depend on github.com/operator-framework/helm-operator-plugins. However, because the latter repo is still marked as experimental, and importing it would result in a large number of accidental dependency changes, I have decided to merely copy code over.

I've tried to minimize changes to both the copied code as well as the existing code in this repo. Because the bulk of the code is copied, I don't think replicating tests would be useful, but I can see if I can copy them over if it turns out to be not too much effort.

Description of the change: Fix a memory leak in the Helm operator code.

Motivation for the change: Excessive memory usage of Helm operators (2GB+)

Checklist

If the pull request includes user-facing changes, extra documentation is required:

• Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
• Add or update relevant sections of the docs website in website/content/en/docs

[everettraven]

@misberner Thanks for the contribution! Apologies for taking so long to get around to reviewing this, but the changes look good to me.

I'd like to get another set of eyes on this prior to merging. @varshaprasad96 would you mind taking a look at this?

[jmccormick2001]

FYI, I think this one is hitting us on one of our customer deployments, infinidat CSI driver which is helm based.

[jberkhahn]

This looks reasonable, but I would prefer to have the tests ported over. They'll get run in our CI, and they'll be deleted along with this stuff when we move over to actually using helm-operator-plugins.

[tuxtof]

ok make some tests with the patch on a 70 nodes clusters trying to deploy 8600 pods to check behaviour

green = pod numbers ( with a scale factor to see it ont he graph)
yellow = operator memory usage

without the patch my operator consume almost 1.2GB of memory

with the patch there is clear improvement for sure less than 800MB

but in both case consumption increase is fully linear with number of pods, which I am rather uncomfortable with

if i disable the watchDependentResources i come back to a pretty more light and stable value less than 200MB

and not sure to understood the impact on the remediation loop , doc is unclear on that

[mpryc]

Hello,

Is there any ETA on this review to be merged? I am hitting the issue where our operator is consuming 3G+ of memory, which I am guessing this PR will resolve.

[jberkhahn]

dunno where OP is, i'll merge this and add the tests myself