-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Helm: Use informer to list helm secrets to improve performance #6354
Conversation
Could you point me to the file where the watch for Secrets in
Yes, wrapping the |
Helm stores its state in secrets inside the cluster. Instead of listing these secrets before every reconciliation of every release, we use an informer to query a local secrets list. This significantly reduced the load on the kubernetes apiserver and etcd Signed-off-by: Luca Berneking <l.berneking@mittwald.de>
Any new updates? Somehow @joelanford was automatically removed as a reviewer when "re-requesting" the review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Lucaber Thanks for the contribution! Apologies for this PR getting lost in the weeds of other PRs. I'll try to get this pushed through.
/lgtm
Description of the change:
Helm stores its state in secrets inside the cluster.
Instead of listing these secrets before every reconciliation of every release, we use an informer to query a local secrets list.
Whats the current status of the helm-operator-plugins repo, should I also create a MR there?
Motivation for the change:
We are running 2 helm-operators with 50 CRs each in a namespace with over 1000 secrets.
Listing these secrets, even with a filter, takes more than 1 second.
Running multiple of these list queries in parallel generates a high load on the kubernetes apiserver and etcd.
Combined with the default reconcile period of 1m this results in a constant high load.
Before and After metrics (after installing this MR on the first and second helm-operator in the cluster):
![image](https://user-images.githubusercontent.com/15067830/224324161-04e0ac2c-925f-45d1-b5f3-2ed4ec3b5ff2.png)
![image](https://user-images.githubusercontent.com/15067830/224323573-5762a896-18cd-40cb-b26a-363f7784348f.png)
This issue was primarily noticed due to a very high load and network traffic between the kubernetes etcd instances.
Checklist
If the pull request includes user-facing changes, extra documentation is required:
changelog/fragments
(seechangelog/fragments/00-template.yaml
)website/content/en/docs