You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Minor chart version updates will always roll opensearch pods even where there is no technical reason to do so. This can be disruptive and time-consuming for large clusters.
Describe the solution you'd like
Care should be taken to only include labels and other data in the statefulset template that are actually necessary to ensure the pods are deployed correctly. Minor chart updates that might change an Ingress (for example) really do not need to be rolling an entire cluster.
For instance, instead of checksumming the entire ConfigMap (which includes labels that will change with every chart version), why not just checksum data in the ConfigMap?
The security config also doesn't seem to require rolling whenever it changes, since to apply those change you still need to run the securityadmin.sh tool manually:
Similarly, the labels on the statefulset pods include the chart version, etc, so that will always roll even if nothing else changed with the configuration or image, etc:
[Triage]
I agree with you @briend, the restart for minor chart should only happen if there is an actual change, however please correct me If i'm wrong today it does rolling restart right, It should not impact the cluster traffic. However I like your idea, is there a way you can create a PR and add the changes your are suggesting ?
Thank you @peterzhuamazon@getsaurabh02
however please correct me If i'm wrong today it does rolling restart right, It should not impact the cluster traffic.
We have some large clusters that can take days to roll. We use a custom readiness probe to avoid rolling the next pod while the cluster is in a non-green state, which slows us down a lot. We might have something to contribute here: #474
Is your feature request related to a problem? Please describe.
Minor chart version updates will always roll opensearch pods even where there is no technical reason to do so. This can be disruptive and time-consuming for large clusters.
Describe the solution you'd like
Care should be taken to only include labels and other data in the statefulset template that are actually necessary to ensure the pods are deployed correctly. Minor chart updates that might change an Ingress (for example) really do not need to be rolling an entire cluster.
For instance, instead of checksumming the entire ConfigMap (which includes labels that will change with every chart version), why not just checksum
data
in the ConfigMap?helm-charts/charts/opensearch/templates/statefulset.yaml
Line 65 in a018f83
The security config also doesn't seem to require rolling whenever it changes, since to apply those change you still need to run the
securityadmin.sh
tool manually:https://opensearch.org/docs/latest/security/configuration/security-admin
We could probably just remove it:
helm-charts/charts/opensearch/templates/statefulset.yaml
Lines 67 to 69 in a018f83
Similarly, the labels on the statefulset pods include the chart version, etc, so that will always roll even if nothing else changed with the configuration or image, etc:
helm-charts/charts/opensearch/templates/statefulset.yaml
Line 58 in a018f83
Perhaps just using
opensearch.selectorLabels
(which are stable across chart versions) would be a good compromiseI can submit an MR if this all sounds reasonable.
The text was updated successfully, but these errors were encountered: