-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[kube-prometheus-stack] Long term fix for kube-prometheus-stack release size issues #3548
Comments
Hi @jkroepke, thank you for this great explanation and discovery process. I'm in favour of moving the CRDs to a sub-chart, it's easier to manage than having to update them manually as we have to do now. Regarding FluxCD, unfortunately I'm not a user so I don't have any experience with it. |
I did some digging around, it doesn't seem like helm has caught up with Server-Side Apply features in Kubernetes. |
@GMartinez-Sisti do you talk about an dedicated helm chart, like https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds or an „unmanaged“ sub-chart like #3547? Mention that we can not use https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds in this context, since the CRDs must be still on the crds directory here |
I think having both a chart for CRDs and an unmanaged chart for CRDs can be kind of confusing, but I get the limitations we have. Looks like an unmanaged chart is currently our best option. |
We need the managed chart for applying the CRDs using ArgoCD since we need to add annotations to tell it to use server side apply, and we cannot currently turn on server side apply globally due to some missing features. |
But we could also add the annotations on kube-prometheus-stack as well, too?
|
This was discussed back when the large description were added and the preferred install method was switched to use server side apply, and iirc the community decision was then that product-specific annotations would not be added by default. Then the new standalone chart was created with the templated crds so we can add our own annotations as needed. We are fine with deprecating the prometheus operator crds chart if the argocd annotation is added to the crds from this new unmanaged chart if the community opinion is now changed. |
We also can everything leave as it is and "just" copy the CRDS from The "unmanged" sub chart |
So |
Correct, yes. |
#3547 is opened for an review. More reviewer may help to guarantee that the change itself does not break something. |
I could be misunderstanding but seems this is a breaking change. Checking out the latest version of the chart with Are we expected to also install or track the CRDs separately as another chart? |
This was a breaking change that's why the major version number was increased. The upgrade doc specifies what you need to do for this upgrade. |
Thanks re-reading I had only seen the |
On my machine, it works fine
|
We tried updating while deploying with Cloning it down and running
|
What about
Maybe the FluxCD integration is not compatible with the approach here. Feels like an issue from FluxCD that CRDs from sub-charts are not supported, not sure. But from Helm and Helm Chart point of view, there isn't a breaking change. |
Fluxcd is compatible with that approach. It seems our issue was the |
Update to latest kube-prometheus-stack chart version. Changes: 55.x This version upgrades Prometheus-Operator to v0.70.0. 54.x Grafana Helm Chart has bumped to version 7. 53.x This version upgrades Prometheus-Operator to v0.69.1, Prometheus to 2.47.2. 52.x This includes the ability to select between using existing secrets or create new secret objects for various thanos config. The defaults have not changed but if you were setting: - `thanosRuler.thanosRulerSpec.alertmanagersConfig` or - `thanosRuler.thanosRulerSpec.objectStorageConfig` or - `thanosRuler.thanosRulerSpec.queryConfig` or - `prometheus.prometheusSpec.thanos.objectStorageConfig` you will have to need to set `existingSecret` or `secret` based on your requirement. 51.x This version upgrades Prometheus-Operator to v0.68.0, Prometheus to 2.47.0 and Thanos to v0.32.2. 50.x This version requires Kubernetes 1.19+. We do not expect any breaking changes in this version. 49.x This version upgrades Prometheus-Operator to v0.67.1, 0, Alertmanager to v0.26.0, Prometheus to 2.46.0 and Thanos to v0.32.0. 48.x This version moved all CRDs into a dedicated sub-chart. No new CRDs are introduced in this version. See <prometheus-community/helm-charts#3548> for more context. We do not expect any breaking changes in this version. 47.x This version upgrades Prometheus-Operator to v0.66.0 with new CRDs (PrometheusAgent and ScrapeConfig). 46.x This version upgrades Prometheus-Operator to v0.65.1 with new CRDs (PrometheusAgent and ScrapeConfig), Prometheus to v2.44.0 and Thanos to v0.31.0. 45.x This version upgrades Prometheus-Operator to v0.63.0, Prometheus to v2.42.0 and Thanos to v0.30.2.
Update to latest kube-prometheus-stack chart version. Changes: 55.x This version upgrades Prometheus-Operator to v0.70.0. 54.x Grafana Helm Chart has bumped to version 7. 53.x This version upgrades Prometheus-Operator to v0.69.1, Prometheus to 2.47.2. 52.x This includes the ability to select between using existing secrets or create new secret objects for various thanos config. The defaults have not changed but if you were setting: - `thanosRuler.thanosRulerSpec.alertmanagersConfig` or - `thanosRuler.thanosRulerSpec.objectStorageConfig` or - `thanosRuler.thanosRulerSpec.queryConfig` or - `prometheus.prometheusSpec.thanos.objectStorageConfig` you will have to need to set `existingSecret` or `secret` based on your requirement. 51.x This version upgrades Prometheus-Operator to v0.68.0, Prometheus to 2.47.0 and Thanos to v0.32.2. 50.x This version requires Kubernetes 1.19+. We do not expect any breaking changes in this version. 49.x This version upgrades Prometheus-Operator to v0.67.1, 0, Alertmanager to v0.26.0, Prometheus to 2.46.0 and Thanos to v0.32.0. 48.x This version moved all CRDs into a dedicated sub-chart. No new CRDs are introduced in this version. See <prometheus-community/helm-charts#3548> for more context. We do not expect any breaking changes in this version. 47.x This version upgrades Prometheus-Operator to v0.66.0 with new CRDs (PrometheusAgent and ScrapeConfig). 46.x This version upgrades Prometheus-Operator to v0.65.1 with new CRDs (PrometheusAgent and ScrapeConfig), Prometheus to v2.44.0 and Thanos to v0.31.0. 45.x This version upgrades Prometheus-Operator to v0.63.0, Prometheus to v2.42.0 and Thanos to v0.30.2.
Hi @prometheus-community/helm-charts-maintainers,
I would like to invite some of you to collect some ideas to solve the issues around the release size of kube-prometheus-stack.
#3083 introduced the issue again, which could the resolved once more by minify the dashboards again.
However, if Prometheus Operator increases the sizes of the CRDs again, we may hit this issue soon.
I do some analysis of the current release data, based on #3546. Here are the steps that I do to gain some reproducable resultes:
By enable the windows monitoring feature from #3083, the release size is
1031276
(release name: kube-prometheus-stack / namespace: default). Without the windows monitoring, it's1025044
.Both values are close to the hard-limit of
1048576
bytes.To gain the full raw data of the helm release, run
This give us the raw JSON data of the helm release. Then I upload the data to an json size analyser:
Most of the release size is files section:
Add md files to helmignore.
As we see, the README.md is included which is not necessary. I did a test with exclude the 2 README files (add them to the .helmignore file) which only sizes 11KB here
1031276
->1010524
0.9% savings
Minify CRD files.
The size of the CRDs are pretty huge. In theory, a lot of data could be saved by store them as minified JSON, too.
Minify all YAML files
Would result into a much lower release size.
1031276
->851536
bytes17.4% savings.
Risks: Unknown behavior in ArgoCD/FluxCD.
Move CRD files into a sub-chart.
After reading helm/helm#11493, it seems like that helm has a different behavior for sub-charts. We all love Helm.
I tried this out by moving the CRDs to an unmanaged sub-chart, named
kube-prometheus-stack-crds
:See #3547, for an POC.
Results:
Running
helm install
still install all CRDs on a fresh installation. However, they are not longer included into the helm release. This is fine, since helm does not manage CRD anyways.The CRD files are still included into the chart package (
helm package
), which is not the case by using the .helmignore file.As result, the total release size after moving the CRD is reduced now:
1031276
->392092
bytes61% savings.
My proposal would be to continue the POC from #3547. I would like to see some things from some maintainers. I learned that FluxCD has a special handling for CRDs and it would be interest into it, if this approach is still compatible with FluxCD.
This would finally resolve the
Too long: must have at most 1048576 bytes
issue forever and we have tons of new space for new manifests. If kube-prometheus-stack hits the release limit on some point in the future again, we could think about todo the same for the dashboard manifests. But for now, moving the CRDs into a sub-chart looks like the easiest one.The text was updated successfully, but these errors were encountered: