Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task lifecycle restart #14127

Merged
merged 14 commits into from
Aug 24, 2022
Merged

Task lifecycle restart #14127

merged 14 commits into from
Aug 24, 2022

Commits on Aug 22, 2022

  1. allocrunner: handle lifecycle when all tasks die

    When all tasks die the Coordinator must transition to its terminal
    state, coordinatorStatePoststop, to unblock poststop tasks. Since this
    could happen at any time (for example, a prestart task dies), all states
    must be able to transition to this terminal state.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    0b0fcde View commit details
    Browse the repository at this point in the history
  2. allocrunner: implement different alloc restarts

    Add a new alloc restart mode where all tasks are restarted, even if they
    have already exited. Also unifies the alloc restart logic to use the
    implementation that restarts tasks concurrently and ignores
    ErrTaskNotRunning errors since those are expected when restarting the
    allocation.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    7e3efae View commit details
    Browse the repository at this point in the history
  3. allocrunner: allow tasks to run again

    Prevent the task runner Run() method from exiting to allow a dead task
    to run again. When the task runner is signaled to restart, the function
    will jump back to the MAIN loop and run it again.
    
    The task runner determines if a task needs to run again based on two new
    task events that were added to differentiate between a request to
    restart a specific task, the tasks that are currently running, or all
    tasks that have already run.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    d704f84 View commit details
    Browse the repository at this point in the history
  4. api/cli: add support for all tasks alloc restart

    Implement the new -all-tasks alloc restart CLI flag and its API
    counterpar, AllTasks. The client endpoint calls the appropriate restart
    method from the allocrunner depending on the restart parameters used.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    be62529 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e9a4035 View commit details
    Browse the repository at this point in the history
  6. allocrunner: kill taskrunners if all tasks are dead

    When all non-poststop tasks are dead we need to kill the taskrunners so
    we don't leak their goroutines, which are blocked in the alloc restart
    loop. This also ensures the allocrunner exits on its own.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    34be197 View commit details
    Browse the repository at this point in the history
  7. taskrunner: fix tests that waited on WaitCh

    Now that "dead" tasks may run again, the taskrunner Run() method will
    not return when the task finishes running, so tests must wait for the
    task state to be "dead" instead of using the WaitCh, since it won't be
    closed until the taskrunner is killed.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    275819a View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    0781b75 View commit details
    Browse the repository at this point in the history
  9. changelog: add entry for #14127

    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    4400c37 View commit details
    Browse the repository at this point in the history
  10. taskrunner: fix restore logic.

    The first implementation of the task runner restore process relied on
    server data (`tr.Alloc().TerminalStatus()`) which may not be available
    to the client at the time of restore.
    
    It also had the incorrect code path. When restoring a dead task the
    driver handle always needs to be clear cleanly using `clearDriverHandle`
    otherwise, after exiting the MAIN loop, the task may be killed by
    `tr.handleKill`.
    
    The fix is to store the state of the Run() loop in the task runner local
    client state: if the task runner ever exits this loop cleanly (not with
    a shutdown) it will never be able to run again. So if the Run() loops
    starts with this local state flag set, it must exit early.
    
    This local state flag is also being checked on task restart requests. If
    the task is "dead" and its Run() loop is not active it will never be
    able to run again.
    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    b97b122 View commit details
    Browse the repository at this point in the history
  11. address code review requests

    lgfa29 committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    c56cfdb View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2022

  1. apply more code review changes

    lgfa29 committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    647f071 View commit details
    Browse the repository at this point in the history
  2. taskrunner: add different Restart modes

    Using the task event to differentiate between the allocrunner restart
    methods proved to be confusing for developers to understand how it all
    worked.
    
    So instead of relying on the event type, this commit separated the logic
    of restarting an taskRunner into two methods:
    - `Restart` will retain the current behaviour and only will only restart
      the task if it's currently running.
    - `ForceRestart` is the new method where a `dead` task is allowed to
      restart if its `Run()` method is still active. Callers will need to
      restart the allocRunner taskCoordinator to make sure it will allow the
      task to run again.
    lgfa29 committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    e620cd1 View commit details
    Browse the repository at this point in the history
  3. minor fixes

    lgfa29 committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    8037a17 View commit details
    Browse the repository at this point in the history