Add default readiness check for webhook server?

[ncdc]

In the Cluster API project, we're trying to automate the following in our CI testing:

1. Create a kind cluster
2. Deploy cert-manager
3. Wait for cert-manager to be ready
4. Deploy Cluster API (CRDs, RBAC, Deployments, etc)
5. Wait for Cluster API pods to be ready (so our webhooks are ready)
6. Create a Cluster, Machine, etc.

When we're waiting for the Cluster API pods to be ready, we've noticed some test flakes because it appears the pods are ready but their webhook servers are not fully online and ready to respond to requests yet.

I've thought about adding a readiness probe to our Manager, but I believe there is still a chance that it could return a false positive; i.e., the check returns ready, but because the webhook server is technically a different server than the readiness server, the webhook server might not be ready yet.

Would it make sense to add a default readiness check for the webhook server?

[DirectXMan12]

yeah, that makes sense

[DirectXMan12]

/kind feature
/priority important-longterm

[ncdc]

@DirectXMan12 I can work on this. How much flexibility would you like for the check? Should we add AddReadyzCheck() to webhook.Server? Or start simple and hard-code a simple ping response?

[vincepri]

/help

[k8s-ci-robot]

@vincepri:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ncdc]

@vincepri I'm still interested in working on this. Do you have thoughts on my question above?

[vincepri]

I'd start simple and hardcode the Readyz checks for now, we can provide a flag later. This should also be backward compatible

[ncdc]

Define backward compat here?

[vincepri]

As in, adding new endpoints for readiness is additive and doesn't impact the current behaviors of the webhook server

[phoracek]

Just curious, what is this supposed to on top of #419 ? Just to use the probe by default? It seems fairly simple to add a readiness probe in the current state of the controller-runtime.

[ncdc]

This would just be exposing checks that people could configure in their pods. Not sure there's any compat issues or impacts.

@phoracek please see the description above for the rationale.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[alenkacz]

@ncdc are you still working on this? I've just spent many hours trying to figure out a nice way how to create such readiness check from outside that I am starting to think it would be easier to implement it within controller-runtime

[ncdc]

@alenkacz not at the moment, no