Nomad Tasks with Restart "delay" should never get into the case of TaskNotRestarting

[camerondavison]

The logic

nomad/client/restarts.go

Line 155 in 23fc2f8

if r.policy.Mode == structs.RestartPolicyModeFail {

that exists for handleWaitResult should also be applied to handleStartError

nomad/client/restarts.go

Line 134 in 23fc2f8

if r.count > r.policy.Attempts {

Also this includes

nomad/client/restarts.go

Line 129 in 23fc2f8

if rerr, ok := r.startErr.(*cstructs.RecoverableError); !(ok && rerr.Recoverable) {

things deemed ReasonUnrecoverableErrror. If the policy is set to Restart "delay" instead of "fail" I feel like there should not be anything that is considered Unrecoverable unless it is some kind of task definition error.

I am not really able to create a list of unrecoverable errors. Looks like the default for a start error is unrecoverable, and then you specify some as recoverable. Some things that I have seen marked as unrecoverable are

• unable to find image in registry (Error while pulling image: Get ******: dial tcp *****: connect: network is unreachable, certainly recoverable)
• unable to bind to port (again recoverable, especially if it makes it pick a new random port)

[diptanu]

@a86c6f7964 Yeah, we could apply the logic of restarting tasks which have exhausted the restart counts within the restart-interval and are failing because of recoverable errors.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.