-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad Job Consul Service Registration Flapping in v0.8.3 #4358
Comments
I ensured the fix that @apenney put in to ensure the Nomad client is connecting to consul over 127.0.0.1 before raising this |
@nathanwebsterdotme I wasn't able to reproduce this in a single node consul cluster running 1.1.0. Had some ansible errors trying your setup that I need to work through. When your Consul cluster is up, is it stable? I am wondering if there are other issues with the consul cluster in your test setup like a serf health failure that is causing the local agent to fail its checks, and the only reason that Nomad v0.8.1 works is because it did not sync services correctly until PR #4170 |
Consul cluster is stable and the service has been registered successfully for over 24 hours now using Nomad v0.8.1. |
Test binary nomad.zip |
@nathanwebsterdotme this should be fixed in the 0.8.4 release candidate. Thanks for reporting this. |
Just FYI, I ran into the same problem yesterday when upgrading our cluster. We run both server and client on the same nodes and reverting to 0.8.1 did the trick. Thanks for @nathanwebsterdotme for providing all the info needed to debug and @preetapan for the patch and the info on when the fix will be available :) |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
consul: v1.1.0
nomad: v0.8.3
Operating system and Environment details
ubuntu 16.04
Issue
When testing a simple local stack of Consul and Nomad on Vagrant, I am seeing the service registered in consul via a nomad job flapping (registers then deregisters constantly). This is not health check related, but the service is constantly being deregistered.
This seems to have been introduced in 0.8.3 as the issue doesn't happen in 0.8.1
Same issue as #4256
Reproduction steps
Changing the Nomad version from v0.8.3 to v0.8.1 fixes the problem and the service stays registered in consul correctly.
Broken: https://github.com/nathanwebsterdotme/hashicorp_stack/tree/612b4375d6dc2dbceaa7d685e02ad6027eb46347 . (you can clone, run vagrant up then run the nomad job from nomad1 if you want to re-create).
Working: (version of Nomad changed to v.0.8.1)
https://github.com/nathanwebsterdotme/hashicorp_stack/tree/9a37c4183c04c881bf36bbd0845fa72e1fdd020a
Job file
https://github.com/nathanwebsterdotme/hashicorp_stack/blob/master/nomad/jobs/simpleweb.nomad
The text was updated successfully, but these errors were encountered: