-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Task lifecycle restart #14127
Task lifecycle restart #14127
Commits on Aug 22, 2022
-
allocrunner: handle lifecycle when all tasks die
When all tasks die the Coordinator must transition to its terminal state, coordinatorStatePoststop, to unblock poststop tasks. Since this could happen at any time (for example, a prestart task dies), all states must be able to transition to this terminal state.
Configuration menu - View commit details
-
Copy full SHA for 0b0fcde - Browse repository at this point
Copy the full SHA 0b0fcdeView commit details -
allocrunner: implement different alloc restarts
Add a new alloc restart mode where all tasks are restarted, even if they have already exited. Also unifies the alloc restart logic to use the implementation that restarts tasks concurrently and ignores ErrTaskNotRunning errors since those are expected when restarting the allocation.
Configuration menu - View commit details
-
Copy full SHA for 7e3efae - Browse repository at this point
Copy the full SHA 7e3efaeView commit details -
allocrunner: allow tasks to run again
Prevent the task runner Run() method from exiting to allow a dead task to run again. When the task runner is signaled to restart, the function will jump back to the MAIN loop and run it again. The task runner determines if a task needs to run again based on two new task events that were added to differentiate between a request to restart a specific task, the tasks that are currently running, or all tasks that have already run.
Configuration menu - View commit details
-
Copy full SHA for d704f84 - Browse repository at this point
Copy the full SHA d704f84View commit details -
api/cli: add support for all tasks alloc restart
Implement the new -all-tasks alloc restart CLI flag and its API counterpar, AllTasks. The client endpoint calls the appropriate restart method from the allocrunner depending on the restart parameters used.
Configuration menu - View commit details
-
Copy full SHA for be62529 - Browse repository at this point
Copy the full SHA be62529View commit details -
Configuration menu - View commit details
-
Copy full SHA for e9a4035 - Browse repository at this point
Copy the full SHA e9a4035View commit details -
allocrunner: kill taskrunners if all tasks are dead
When all non-poststop tasks are dead we need to kill the taskrunners so we don't leak their goroutines, which are blocked in the alloc restart loop. This also ensures the allocrunner exits on its own.
Configuration menu - View commit details
-
Copy full SHA for 34be197 - Browse repository at this point
Copy the full SHA 34be197View commit details -
taskrunner: fix tests that waited on WaitCh
Now that "dead" tasks may run again, the taskrunner Run() method will not return when the task finishes running, so tests must wait for the task state to be "dead" instead of using the WaitCh, since it won't be closed until the taskrunner is killed.
Configuration menu - View commit details
-
Copy full SHA for 275819a - Browse repository at this point
Copy the full SHA 275819aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0781b75 - Browse repository at this point
Copy the full SHA 0781b75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4400c37 - Browse repository at this point
Copy the full SHA 4400c37View commit details -
taskrunner: fix restore logic.
The first implementation of the task runner restore process relied on server data (`tr.Alloc().TerminalStatus()`) which may not be available to the client at the time of restore. It also had the incorrect code path. When restoring a dead task the driver handle always needs to be clear cleanly using `clearDriverHandle` otherwise, after exiting the MAIN loop, the task may be killed by `tr.handleKill`. The fix is to store the state of the Run() loop in the task runner local client state: if the task runner ever exits this loop cleanly (not with a shutdown) it will never be able to run again. So if the Run() loops starts with this local state flag set, it must exit early. This local state flag is also being checked on task restart requests. If the task is "dead" and its Run() loop is not active it will never be able to run again.
Configuration menu - View commit details
-
Copy full SHA for b97b122 - Browse repository at this point
Copy the full SHA b97b122View commit details -
Configuration menu - View commit details
-
Copy full SHA for c56cfdb - Browse repository at this point
Copy the full SHA c56cfdbView commit details
Commits on Aug 24, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 647f071 - Browse repository at this point
Copy the full SHA 647f071View commit details -
taskrunner: add different Restart modes
Using the task event to differentiate between the allocrunner restart methods proved to be confusing for developers to understand how it all worked. So instead of relying on the event type, this commit separated the logic of restarting an taskRunner into two methods: - `Restart` will retain the current behaviour and only will only restart the task if it's currently running. - `ForceRestart` is the new method where a `dead` task is allowed to restart if its `Run()` method is still active. Callers will need to restart the allocRunner taskCoordinator to make sure it will allow the task to run again.
Configuration menu - View commit details
-
Copy full SHA for e620cd1 - Browse repository at this point
Copy the full SHA e620cd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8037a17 - Browse repository at this point
Copy the full SHA 8037a17View commit details