startupProbe behaviour changed in v1.21.0

[ykulazhenkov]

What happened:

We detected startupProbe behavior change in v1.21.0

Example:
POD has startupProbe and livenessProbe for some containers

Behavior in v1.20.5:

1. POD starts
2. startupProbe applied
3. if startupProbe success, then start checks for livenessProbe
4. POD working
5. livenessProbe failed
6. container restarted
7. startupProbe applied
8. if startupProbe success, then start checks for livenessProbe
9. POD working

Behavior in v1.21.0:

1. POD starts
2. startupProbe applied
3. if startupProbe success, then start checks for livenessProbe
4. POD working
5. livenessProbe failed
6. container restarted
7. livenessProbe applied, startupProbe skipped
8. POD not ready yet, killed by liveness check
9. POD in restart loop

So, in v1.21 startupProbe is applied to "fresh" starts only and not used when container restarted.

What you expected to happen:

There is not mentions in changelog that behavior of the startupProbe is changed.
startupProbe should run on container restarts also.

How to reproduce it (as minimally and precisely as possible):

1. Create POD with startupProbe and livenessProbe
2. Make livenessProbe fail
3. Check that startupProbe doesn't apply after container restart

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: baremetal

[k8s-ci-robot]

@ykulazhenkov: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ykulazhenkov]

/sig node

[wzshiming]

/assign

[matthyx]

Thanks for the quick fix @wzshiming

[neolit123]

it was fixed in master (1.22) but not for 1.21, does it need a backport?
/reopen

[k8s-ci-robot]

@neolit123: Reopened this issue.

In response to this:

it was fixed in master (1.22) but not for 1.21, does it need a backport?
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/close

never mind i just saw the backports.

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

/close

never mind i just saw the backports.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

can you summarize the impacted patch releases (looks like the regression was backported to 1.18-1.20 as well)

[wzshiming]

These 4 patch releases are impacted, 1.18.18, 1.19.10, 1.20.6, 1.21.0
#101226
#101225
#101224
#101223