Task is started despite skipped pre-start hooks #18659
Labels
stage/accepted
Confirmed, and intend to work on. No timeline committment though.
theme/client
type/bug
Projects
Nomad version
Tip
Operating system and Environment details
Linux
Issue
There is a race between allocation termination and task startup that can cause the task to start without the pre-start hooks having been executed. This may cause the driver to fail during startup because the task directory is not initialized. We suspect other failure modes are possible here (eg. the task directory may exist while re-discovering allocations during agent restart but other hooks don’t run), but the driver failure has arisen several times.
Details
The issue arose with the driver failing with the following sequence of events:
While this involved a leader and sidecar task, the error can occur with a single task.
Verify that skipping the pre-start hooks causes the observed driver failure
Patch nomad to unconditionally skip pre-start hooks.
When attempting to start a simple task,job the allocation fails with:
Finding the root cause of the raciness
The prestart hooks are only skipped when tr.sholdShutdown() is true.
nomad/client/allocrunner/taskrunner/task_runner_hooks.go
Lines 195 to 201 in 73e372a
Hence, for the prestart hooks to be skipped and yet the task still run, it must be the case that:
nomad/client/allocrunner/taskrunner/task_runner.go
Lines 570 to 614 in 73e372a
For tr.shouldShutdown() to become true, tr.Alloc must have transitioned from non-terminal status to terminal status before
nomad/client/allocrunner/taskrunner/task_runner.go
Lines 734 to 746 in 73e372a
The state of tr.Alloc is updated in
nomad/client/allocrunner/taskrunner/task_runner.go
Lines 1410 to 1411 in 73e372a
called from
nomad/client/allocrunner/alloc_runner.go
Lines 956 to 995 in 73e372a
which is called as a goroutine
nomad/client/allocrunner/alloc_runner.go
Lines 321 to 330 in 3d63bc6
In handleAllocUpdate, the state of the task runner is first updated and then tasks are killed (closing KillCtx). However, because handleAllocUpdate runs on a different goroutine as TaskRunner::Run, the following ordering of events is possible and would result in the observed behavior:
Reproduction steps
Without any other patches applied, add sleep statements to widen the window for the race to trigger.
Then,
Expected Result
When the allocation is known to be terminal and pre-start hooks are skipped, we don’t proceed to start the task.
Actual Result
Job file (if appropriate)
The task is started without a properly prepared runtime environment, causing the driver to fail.
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
[0]
The text was updated successfully, but these errors were encountered: