HNC slow to become Ready on 1.21 and earlier

[adrianludwin]

I've found that HNC becomes Ready on 1.22 after about 10s, while on 1.20 and 1.21 it takes about 80+ seconds. Adding a Readiness probe seems to solve the problem - I'm not sure why, since "no readiness probe" is supposed to be equivalent to "always ready" and this does seem to be the case in 1.22. But I can't find any evidence of a bug that got fixed in 1.22 and I'm surprised that something this obviously wrong would have survived so long.

Either way, the solution seems pretty straightforward and I'll make the fix today.

/cc @erikgb

[adrianludwin]

This looks vaguely related to kubernetes/kubernetes#101064 but that bug was introduced in 1.21 (whereas I'm seeing the same behaviour in 1.20) and besides, it doesn't seem to be the same issue.

[adrianludwin]

Hmm, kubernetes/kubernetes#98376 looks possibly related... except that went into 1.21 🤷

[adrianludwin]

kubernetes/kubernetes#101738 looks like a more likely candidate - introduced in 1.22, designed to fix kubernetes/kubernetes#99979. It explicitly was not backported to 1.21 (kubernetes/kubernetes#102681 (comment)).

[adrianludwin]

/cc @BenTheElder

... in case you care :)

[adrianludwin]

Note that I verified that the healthz and readyz endpoints were actually working well this entire time. E.g. I started a busybox, pinged the endpoints directly, and verified that they returned ok long before the containers actually became Ready. The fact that adding a readiness probe with no change to the container fixes the problem is more evidence IMO that this is a workaround to a K8s-side problem.