Nomad marks deployments as successful, before Consul checks are passed. #15536
Comments
|
@shoenig Hi! Could you please look at this issue? |
|
Hi folks, we've marked this issue for triage. |
|
Hi, any updates here? Unfortunately, we can't use this version in production and downgrades are not the strong side of nomad. Do you need more details? |
|
Hi folks, sorry for missing this issue before. I am able to reproduce fairly easily by fudging our use of Consul API to pretend like checks not being registered on time. Will find a fix and backport in the next bugfix release set. |
shoenig
added a commit
that referenced
this issue
Jan 19, 2023
This PR fixes a bug where Nomad assumed any registered Checks would exist in the service registration coming back from Consul. In some cases, the Consul may be slow in processing the check registration, and the response object would not contain checks. Nomad would then scan the empty response looking for Checks with failing health status, finding none, and then marking a task/alloc as healthy. In reality, we must always use Nomad's view of what checks should exist as the source of truth, and compare that with the response Consul gives us, making sure they match, before scanning the Consul response for failing check statuses. Fixes #15536
shoenig
added a commit
that referenced
this issue
Jan 19, 2023
This PR fixes a bug where Nomad assumed any registered Checks would exist in the service registration coming back from Consul. In some cases, the Consul may be slow in processing the check registration, and the response object would not contain checks. Nomad would then scan the empty response looking for Checks with failing health status, finding none, and then marking a task/alloc as healthy. In reality, we must always use Nomad's view of what checks should exist as the source of truth, and compare that with the response Consul gives us, making sure they match, before scanning the Consul response for failing check statuses. Fixes #15536
shoenig
added a commit
that referenced
this issue
Jan 19, 2023
* consul: correctly understand missing consul checks as unhealthy This PR fixes a bug where Nomad assumed any registered Checks would exist in the service registration coming back from Consul. In some cases, the Consul may be slow in processing the check registration, and the response object would not contain checks. Nomad would then scan the empty response looking for Checks with failing health status, finding none, and then marking a task/alloc as healthy. In reality, we must always use Nomad's view of what checks should exist as the source of truth, and compare that with the response Consul gives us, making sure they match, before scanning the Consul response for failing check statuses. Fixes #15536 * consul: minor CR refactor using maps not sets * consul: observe transition from healthy to unhealthy checks * consul: spell healthy correctly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Nomad version and environment details
Nomad 1.1.2 -> 1.4.2
Consul 1.9.5
Ubuntu 20.04
Environment configuration:
Servers: 3 nomad servers, consul servers (services register here) + consul servers with ACL.
Clients: we've installed nomad client and consul agent.
Issue
After upgrading Nomad from 1.1.2 to 1.4.2 some deployments are marked as successful, despite that service has not yet been registered with Consul. This problem is reproduced when running several deployments at the same time.
Reproduction steps
Expected Behaviour
All deployments fail.
Actual Behaviour
Few deployments are successful.
Logs for one allocation for successful deployment
Nomad start to process deployment:
Dockerd start container:
Container fail:
Nomad marks deployment as successful:
Container restart due to restart policy:
First Consul check logs for alloc_id=384708cc-5ca5-1e14-ca6b-7017045ee3c7:
The text was updated successfully, but these errors were encountered: