Both tasks marked as unhealthy if only one fails. #9254
Comments
|
Hi @erichulburd! So Nomad is behaving correctly here but the way it's being presented could probably be more clear. Here's a more minimal reproduction for my colleagues on the UI team, so that they can see this for themselves:
jobspecjob "example" {
datacenters = ["dc1"]
update {
health_check = "checks"
min_healthy_time = "10s"
healthy_deadline = "1m"
progress_deadline = "2m"
}
group "webservers" {
network {
port "www1" {
to = 8080
}
port "www2" {
to = 8081
}
}
task "example-1" {
driver = "docker"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "8080", "-h", "${NOMAD_TASK_DIR}"]
ports = ["www1"]
}
template {
data = "<html>ok</html>"
destination = "local/index.html"
}
service {
check {
port = "www1"
type = "http"
path = "/"
interval = "2s"
timeout = "1s"
}
}
resources {
cpu = 500
memory = 256
}
}
task "example-2" {
driver = "docker"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "8081", "-h", "${NOMAD_TASK_DIR}"]
ports = ["www2"]
}
template {
data = "<html>ok</html>"
destination = "local/index.html"
}
# this health check will never pass
service {
check {
port = "www2"
type = "http"
path = "/does-not-exist"
interval = "2s"
timeout = "1s"
}
}
resources {
cpu = 500
memory = 256
}
}
}
}
At this point the allocation status will look like this. Note that we have an The Nomad client is doing the right thing here: the deployment was not unhealthy by the progress deadline, so the whole allocation is marked as unhealthy. The way this is presented in the CLI is a little weird but because we see the |
It still seems weird to me that both tasks get the event? I suppose the event truly belongs to the alloc but there are no alloc events so this is the workaround?
The type is shown if you drill into the task detail page, but we can probably make this clearer on the alloc detail page too. |
|
I'm seeing this behavior when I have a task with Even when the task exits successfully with status code 0 and everything else is fine, the alloc is marked as unhealthy. This seems like a bug to me, or at least something that needs to have a workaround. If I have a task that is meant to do some kind of initialization then immediately exit, that shouldn't block the other tasks from being considered healthy. Edit. It seems like this issue is what I'm describing: #10058 |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.12.5
Operating system and Environment details
MacOS Catalina 10.15.4, but I'm experiencing similar issues running through OpenStack.
Issue
When a task group has 2+ tasks and one fails, both tasks are marked unhealthy with
Task not running for min_healthy_time of 10s by deadlinein the UI (see reproduction steps below). Maybe this is intentional for some good reason, but the UX is confusing regardless.Reproduction steps
Please see source and instructions in erichulburd/consul-nomad-issues.
Job file (if appropriate)
erichulburd/consul-nomad-issues/job.json.
The text was updated successfully, but these errors were encountered: