Manager Shutdown/Cache Establishment low-impact race-thing

[DirectXMan12]

If you end up stopping the manager just after you start it, you end up manifesting a weird shutdown "race" in which you're still in the middle of controller startup/cache establish.

Fortunately as of newer revisions of controller-runtime (0.7x+ AFAICT), this manifests as a "error received after stop sequence was engaged" log line, which is ok but not great (v0.6.x manifests it as a graceful shutdown timeout, which is 😞 ).

It's not fatal, but it'd be nice if there were a way to avoid this, since it tends to show up a bit in certain test scenarios (e.g. manager start in beforeach, some tests that don't actually gate on controller actions).

AFAICT, you can't just wait for cache establishment, cause you'll race with actually having the controllers register themselves.

We could perhaps avoid this with a concept of runnable readiness, and a corresponding way to get the overall manager readiness. Then, you can wait in your test code for readiness to finish (nice side effect: we get a readyz endpoint "for free").

Reproducer: https://gist.github.com/DirectXMan12/6edc4c1d917b121339e2a4324432d9e7

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.