client: ensure task only runs with prestart hooks

[lgfa29]

Since the allocation in the task runner is updated in a separate goroutine, a race condition may happen where the task is started but the prestart hooks are skipped because the allocation became terminal.

Checking for a terminal allocation before proceeding with the task start ensures the task only runs if the prestart hooks are also executed.

Since shouldShutdown() only uses terminal allocation status, it remains true after the first transition, so it's safe to check it again after the prestart hooks as it will never revert to false.

Some other implementations ideas I considered:

1. Move the check for shouldShutdown() from within prestart() to before it is called. I think this would be more or less equivalent to this approach.
2. Create a read lock on the task runner alloc so that shouldShutdown() is guaranteed not to change while prestart() runs and the task starts. This is probably the most "correct" approach, but since shouldShutdown() can only transition from false to true, checking it again after prestart() seems enough.

Closes #18659

[schmichael]

LGTM! Sad that shouldShutdown() really means "should-shutdown-because-alloc-is-terminal" ... I think it would be safe to stuff the killCtx and shutdownCtx checks inside of it as well, but then we have to consider what that would do in the checks that call shouldShutdown(). So this seems like a nice precise fix.

[tgross]

Nice catch!

[lgfa29]

LGTM! Sad that shouldShutdown() really means "should-shutdown-because-alloc-is-terminal" ... I think it would be safe to stuff the killCtx and shutdownCtx checks inside of it as well, but then we have to consider what that would do in the checks that call shouldShutdown(). So this seems like a nice precise fix.

Yeah, I wanted to make this part of the code more "transactional": either the task starts in full or it doesn't start at all. But that would be a much bigger change I think, so I went for the smaller change necessary to fix the problem.