Service not registered in consul when service block is added as job update #9707
Comments
|
Found more issues with this. I've submitted a new job with the service block in place from the start: job "echo" {
datacenters = ["dc1"]
group "web" {
network {
port "http" {
to = 5678
}
}
task "server" {
driver = "docker"
config {
image = "hashicorp/http-echo"
args = ["-text", "hello world"]
ports = ["http"]
}
service {
name = "echo"
port = "http"
}
}
}
}That worked fine, service Then I added a health check to the job spec, and submitted this updated job (only job "echo" {
datacenters = ["dc1"]
group "web" {
network {
port "http" {
to = 5678
}
}
task "server" {
driver = "docker"
config {
image = "hashicorp/http-echo"
args = ["-text", "hello world"]
ports = ["http"]
}
service {
name = "echo"
port = "http"
check {
name = "http port alive"
type = "http"
path = "/"
interval = "30s"
timeout = "2s"
}
}
}
}
}Health check was not added, and instead the error was logged:
Full related nomad log output: |
|
Thanks for reporting, @sickill. Looks like the problem is we try to optimize out the service registration task runner hook if the initial task doesn't have any services. This of course prevents newly defined services from being registered, as this is the hook that handles service registration updates as well. Group level services don't have this bug, as the equivleent group service alloc runner hook is always enabled. |
shoenig
added
the
stage/accepted
Previously, Nomad would optimize out the services task runner hook for tasks which were initially submitted with no services defined. This causes a problem when the job is later updated to include service(s) on that task, which will result in nothing happening because the hook is not present to handle the service registration in the .Update. Instead, always enable the services hook. The group services alloc runner hook is already always enabled. Fixes #9707
Previously, Nomad would optimize out the services task runner hook for tasks which were initially submitted with no services defined. This causes a problem when the job is later updated to include service(s) on that task, which will result in nothing happening because the hook is not present to handle the service registration in the .Update. Instead, always enable the services hook. The group services alloc runner hook is already always enabled. Fixes #9707
|
🎉 |
Previously, Nomad would optimize out the services task runner hook for tasks which were initially submitted with no services defined. This causes a problem when the job is later updated to include service(s) on that task, which will result in nothing happening because the hook is not present to handle the service registration in the .Update. Instead, always enable the services hook. The group services alloc runner hook is already always enabled. Fixes #9707
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.0.1 (c9c68aa)
Consul version
Consul v1.9.0
Revision a417fe510
Operating system and Environment details
I originally stumbled upon this problem on a cluster running on Ubuntu 20.04
I can reproduce it on macOS Catalina 10.15.7 (the steps below)
Issue
When the job has been running without a
servicedefinition, adding aserviceblock and re-submitting the job has no effect - the service is not registered in consul.Logs don't contain any warnings nor errors, it looks like there's no attempt of service registration at all.
Reproduction steps
Start consul:
consul agent -devStart nomad:
nomad agent -devRun the job file below:
nomad run echo.nomadVerify the service has not been registered in consul, as expected.
Uncomment the
serviceblock, and submit the updated job:nomad run echo.nomadVerify the service has not been registered in consul, while it should have been.
I also tested the reverse scenario:
I first submitted a fresh job (previously not existing in nomad cluster) with the service block. It has been successfully registered in consul. Then I removed the
serviceblock from the job spec and re-submitted the job. The service has been properly removed from consul. So it seems only registration is affected by this bug.Job file (if appropriate)
The text was updated successfully, but these errors were encountered: