Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

health: detect failing tasks #7383

Merged
merged 4 commits into from
Mar 25, 2020
Merged

health: detect failing tasks #7383

merged 4 commits into from
Mar 25, 2020

Commits on Mar 22, 2020

  1. tests: add a check for failing service checks 

    Add tests to check for failing or missing service checks in consul
update.
    Mahmood Ali committed Mar 22, 2020
    Configuration menu
    Copy the full SHA
    3719ff3 View commit details
    Browse the repository at this point in the history

  2. health: fail health if any task is pending 

    Fixes a bug where an allocation is considered healthy if some of the
tasks are being restarted and as such, their checks aren't tracked by
consul agent client.

Here, we fix the immediate case by ensuring that an alloc is healthy
only if tasks are running and the registered checks at the time are
healthy.

Previously, health tracker tracked task "health" independently from
checks and leads to problems when a task restarts.  Consider the
following series of events:

1. all tasks start running -> `tracker.tasksHealthy` is true
2. one task has unhealthy checks and get restarted
3. remaining checks are healthy -> `tracker.checksHealthy` is true
4. propagate health status now that `tracker.tasksHealthy` and
`tracker.checksHealthy`.

This change ensures that we accurately use the latest status of tasks
and checks regardless of their status changes.

Also, ensures that we only consider check health after tasks are
considered healthy, otherwise we risk trusting incomplete checks.

This approach accomodates task dependencies well.  Service jobs can have
prestart short-lived tasks that will terminate before main process runs.
These dead tasks that complete successfully will not negate health
status.
    Mahmood Ali committed Mar 22, 2020
    Configuration menu
    Copy the full SHA
    3132176 View commit details
    Browse the repository at this point in the history

  3. health check account for task lifecycle 

    In service jobs, lifecycles non-sidecar task tweak health logic a bit:
they may terminate successfully without impacting alloc health, but fail
the alloc if they fail.

Sidecars should be treated just like a normal task.
    Mahmood Ali committed Mar 22, 2020
    Configuration menu
    Copy the full SHA
    1454af7 View commit details
    Browse the repository at this point in the history

  4. health tracker: account for group service checks

    Mahmood Ali committed Mar 22, 2020
    Configuration menu
    Copy the full SHA
    525623c View commit details
    Browse the repository at this point in the history