-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix wrong control plane discovery, when serf advertised address located in network, unreachable from clients #11895
Conversation
in another network - unreachable from nomad clients(client when discovering controlplane by consul, got serf advirtised address, but must get rpc)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @tantra35!
The situation you've described here seems a little weird to me, because the existing Status.Peers
endpoint should already be returning the RPC endpoints. Here's an example configuration:
advertise {
http = "192.168.56.10"
rpc = "192.168.56.10"
serf = "10.0.2.15"
}
And then if I curl the appropriate endpoint:
$ curl -H "X-Nomad-Token: $NOMAD_TOKEN" -s "http://localhost:4646/v1/status/peers"
["192.168.56.20:4647","192.168.56.10:4647","192.168.56.30:4647"]%
Can you provide a bit more information about the client configuration you have? Maybe that'd help us figure out what the issue is for you.
@tgross Hm just tested on my test stand and as expected
our nomad configs:
On this stand i up only controlplane, without any clients or anything else, just 3 controlplane servers(nomad + consul). If it make sense consul also advertise wan address like this:
If it will be useful i can provide my Vagrant stand |
@tgross I having rebuilded test stand where only nomad installed(consul not installed in that stand), and I don't understand why you get the right result and I don't?
then i try to get
then try to call Status.Peers via REST interface
And imo this is absolutely expected behavior, I've never seen nomad work any other way, Maybe something changed between master and 1.1.10 versions? But its strange because i doesn't see significant changes between between versions which would address these points |
Suppose follow configuration, when we use nomad federation:
A
andB
)10.168.0.0/16
, rpc and http addresses located in private networks, unique for both regions(192.168.102.0/24
for regionA
, and192.168.103.0/24
for regionB
)10.168.0.0/24
A
on first serverB
on first serverIn such configuration nomad clients when discover controlplane by consul will get addresses from network
10.168.0.0/16
, and of course nothing will work, because they can't reach this networkThis patch try fix this situation by adding new RPC method
RpcPeers
which will return private RPC adresses instead of Serf