-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wait and timeout options with Helm-based operator #4102
Comments
This seems like it would be helpful. The one concern I have is that Kubernetes controllers generally don't block, waiting for certain conditions to exist. Instead, they make their observations and changes, and then wait until those changes trigger another reconcile loop. The loop then repeats until steady state is reached. If Helm's This has me thinking about |
@joelanford that will meet my use case. You are right, I don't think the timeout make sense. |
@jfgosselin Cool! Is this something you would be willing to investigate and contribute? One thing I'm not sure about is whether the Helm libraries expose the same checks that the |
This is closely related to #2491. The operator should mark the CR status Ready only after all the resources are ready by k8s's definition of ready. In Helm, there is an interface that implements ready checking of multiple resources https://github.com/helm/helm/blob/9b42702a4bced339ff424a78ad68dd6be6e1a80a/pkg/kube/interface.go#L38-L44 The hooks that perform the checks are here |
@drewwells It looks like For Unfortunately, it looks like the Ideally this block of logic would be extracted and exported into a new |
I agree about the action package, it's very lacking in its flexibility. You're right about What is the objection to using action.Install.Wait flag? I think even using client.Wait here will have limited visibility from within the operator itself. |
@drewwells There are a few reasons I don't think we should use As I mentioned in my first comment, it is generally an anti-pattern to wait in Kubernetes controllers. The more accepted pattern is to observe the current state of the CR and its resources (e.g. ask is the release ready?), make any changes (e.g. set CR status based on whether the resources are ready), and immediately return. Then if/when any of the status blocks of any of the release resources change, our reconciler will be triggered again, and we can again check if the release is ready. This process repeats until all of the release resource status blocks achieve steady state. If the final steady state is that all of the resources are ready, then the reconciler will set the CR status to reflect that. With this design the operator can scale to handle a very large number of CRs and their releases being active at the same time. If the reconciler is allowed to block, the operator will only be able to handle Another benefit of this design is that the question of whether a release is ready is always checked. If steady state was reached and the release became ready, that doesn't mean it will ALWAYS be ready. If a pod terminates or a service loses its endpoints, the reconciler will be triggered and it will change the CR status from ready to not ready. |
We can cheat here just like helm-cli does. It marks the release as ready as soon as it becomes available even when that is only true for a short period. Programs that do not have sufficiently long startupProbes will pass as ready then start crash looping, helm does not catch this. I think here we need to distinguish between operations helm can handle vs trying to micromanage resources of a release. If a subresource is deleted, the operator could sort out how to bring it back. This can be handled by watching subresources for CUD events https://godoc.org/sigs.k8s.io/controller-runtime/pkg/builder#Builder.Owns. The other one is I guess is to update status if the release is in bad health. I’m not so certain operator is responsible for this. Most operators I have used act sort of like k8s replicasets. They will do very simple things like recreate pods when deleted and update statuses if things go wrong. They can’t fix a broken pod and don’t try to. Keeping in this mind, it seems ideal that helm-operator watches all the sub-resources and attempts to make some decisions on them at least until initial ready. It is a nice to have for the continuous watching. I suspect the current behavior which appears to be polling to sync the state over time. If configmap is mutated, requeue is reconciled, lookup resource owner, find the desired state, set the configmap back to expected state. It sounds expensive memory wise to watch this many objects so polling is fine here. |
@joelanford sorry, I've never coded in Go |
@jfgosselin No worries. This is not currently a high priority for the core SDK team maintainers, but it is definitely a nice feature, and we would definitely accept a PR. I think the direction we would want to go is to:
If anyone is interested in taking this work on, let us know! |
BTW not sure how they did it in the Flux Helm Operator but I think they support this feature (we didn't use Flux for some other reasons). |
I didn't look too closely, but it looks like they just allow the |
Workaround,
Current
If there was a Status for "DeployedSuccessful", then users could use kubectl to wait on this status to take action.
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
@openshift-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Feature Request
Describe the problem you need a feature to resolve.
The Helm-based operator does not wait until the Deployments have minimum (Desired minus maxUnavailable) Pods in ready state before marking the release as successful.
Describe the solution you'd like.
Expose a wait and timeout option like the Helm cli and update the CustomResource status accordingly.
--timeout: A value in seconds to wait for Kubernetes commands to complete This defaults to 5m0s
--wait: Waits until all Pods are in a ready state, PVCs are bound, Deployments have minimum (Desired minus maxUnavailable) Pods in ready state and Services have an IP address (and Ingress if a LoadBalancer) before marking the release as successful. It will wait for as long as the --timeout value. If timeout is reached, the release will be marked as FAILED. Note: In scenarios where Deployment has replicas set to 1 and maxUnavailable is not set to 0 as part of rolling update strategy, --wait will return as ready as it has satisfied the minimum Pod in ready condition.
/language helm
The text was updated successfully, but these errors were encountered: