client: always run alloc cleanup hooks on final update

[shoenig]

This PR fixes a bug where alloc pre-kill hooks were not run in the
edge case where there are no live tasks remaining, but it is also
the final update to process for the (terminal) allocation. We need
to run cleanup hooks here, otherwise they will not run until the
allocation gets garbage collected (i.e. via Destroy()), possibly
at a distant time in the future.

Fixes #15477

[schmichael]

I think this will erroneously run prekill hooks on client agent shutdown.

[shoenig]

I added a guard with ar.isShuttingDown()

[gulducat]

if preKillHooks() shouldn't be run during agent shutdown, would it make sense to put the guard inside of preKillHooks() instead of here? more broadly, if this func is a potential foot-shoot, is there a way to make it less foot-shoot-able?

[schmichael]

Good question @gulducat. I'm afraid it needs to be up to the caller to decide whether prekill hooks should run or not because only the callers knows whether a kill event (eg a task dying) was triggered or not. Once your in prekill, a shutdown may be issued concurrently, but that doesn't change the fact that the prekill was triggered by a kill event. So I think this is the right approach as @shoenig's code is able to differentiate a kill event from an agent shutdown event.

That being said every operation in Nomad needs to be crash safe which is why prekill hooks are also run when garbage collecting dead allocations. So prekill should always eventually be called on every terminal allocation regardless of the triggering event.

[lgfa29]

I think I understand things now 😅

preKillHooks is only called by killTasks() and killTasks is called in a few places:

• destroyImpl which is quite expected.
• handleTaskStateUpdates which also makes sense since we may need to kill all tasks if one of the tasks fails.
• handleAllocUpdate which again makes sense since the alloc is stopped we need to kill the tasks.

The part that was confusing me was why handleAllocUpdate was not triggering the preKillHooks: the allocation does eventually die and is marked with ClientStatus: failed.

The reason is that this is a client-only alloc update. When watching allocations the client ignores changes that it already knows about and so the ar.Update() method is not called when the allocation fails client-side.

We don't want to destroy the alloc runner so the only place left to call the hooks is in handleTaskStateUpdates like @shoenig did.

The second thing that was confusing to me is why we only check for killEvent if there are live runners, I feel like we should always be checking that and calling killTasks() if not nil? Something like https://github.com/hashicorp/nomad/compare/wip-luiz-kill-ar (this probably breaks other stuff since task lifecycle is finicky 😅).

All of this to say that I think we can use if killEvent != nil here to guard the preKillHooks so it matches the other conditional.

[gulducat]

💀 ✅