-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad error's on TCP lookups for retry_join entries when IPv6 DNS resolvers are used #1317
Comments
@slackpad An issue resolving IPv6 addresses in memberlist? |
@The-Loeki we are working on some ipv6 issues, but I can't reproduce this one. Is it still an issue with the current master? We built with go 1.7rc3 - maybe it's an issue in (older) go? |
hi, Beware the specs; the DNS resolver is an IPv6 address, which causes this bug, not the Nomad server or any of it's config per se (I got the other bugs for those ;) ) |
Yes, I have one single ipv6 address in the resolv.conf (but I'm running 0.4.1-dev) |
Aighty then, maybe it semi-accidentally got fixxed along with e.g. #1465 et al. ? Unfortunately I have no access to master binaries at this time, but I'll report back on this one as soon as a new release hits the wires. In the meantime the bug seems of little real consequence for now; there's a possibility DNSSEC validation fails if UDP is force-used, but other than that I'll be more than happy to sit it out 'till that new release ;) |
I tried reproducing with this release but it seems to work too ... |
hmmm... That's odd...
|
I tried with a nomad client now (not a server); are you using consul? |
You might be on to something here; looking at the logs it does seem only to happen on server nodes. And yes, same setup (IPv6 only w/socket binds) |
Also on a server no problem. I now have two ipv6 addresses in resolv.conf, running 0.4.0, and have (redacted) log:
|
This is really odd then...
nomad log:
What can I do to help? |
Ok some major differences are:
I'm on holiday from today so may take a while before I can try to reproduce |
Thanks, that was the hint that was needed; the I'm not sure I'm following how this works here; the Consul services don't get registered properly as per #1318 ;) Anyway, we're closer to the source of the bug. I've adjusted the description of this issue to match |
Two reasons come to mind without looking at the code.
First, nomad as a server has a state, in your case in `/var/nomad`; did you
delete its content?
Second, though registration in consul with v0.4.0 is not correct (checks) ,
it is not entirely broken. If you registered with a local consul agent, the
ip may have worked. Try with `dig ANY nomad.service.consul
@yourconsulserver` (substitute where necessary)
Good luck hunting the bug 😉
|
Hi, we're closing this out as there hasn't been any action on this ticket a long time and there have been a lot of changes to Nomad networking since then. If this is still an issue for anybody, please feel free to open a new Issue and link to this one! |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.4.0
Operating system and Environment details
CentOS 7 / Ubuntu 16.04 up-to-date
Issue
When IPv6 resolvers are used in
/etc/resolv.conf
and entries are in theretry_join
config, Nomad will always report:(
2a00:f10:ff04:153::53
is the DNS server in resolv.conf)Reproduction steps
Setup & use IPv6 resolvers, kick off a Nomad cluster and add some servers in the
retry_join
sectionI'm not entirely sure how, but the first part of the error msg seems to stem from
https://github.com/hashicorp/nomad/blob/v0.4.0-rc1/vendor/github.com/hashicorp/memberlist/memberlist.go#L301
and the second part from
https://github.com/golang/go/blob/master/src/net/ipsock.go
The text was updated successfully, but these errors were encountered: