Commits on Feb 28, 2023

1. client: use RPC address and not serf after initial Consul discovery …

   Nomad servers can advertise independent IP addresses for `serf` and
   `rpc`. Somewhat unexpectedly, the `serf` address is also used for both Serf and
   server-to-server RPC communication (including Raft RPC). The address advertised
   for `rpc` is only used for client-to-server RPC. This split was introduced
   intentionally in Nomad 0.8.
   
   When clients are using Consul discovery for connecting to servers, they get an
   initial discovery set from Consul and use the correct `rpc` tag in Consul to get
   a list of adddresses for servers. The client then makes a `Status.Peers` RPC to
   get the list of those servers that are raft peers. But this endpoint is shared
   between servers and clients, and provides the address used for Raft.
   
   Most of the time this is harmless because servers will bind on 0.0.0.0 anyways.,
   But in topologies where servers are on a private network and clients are on
   separate subnets (or even public subnets), clients will make initial contact
   with the server to get the list of peers but then populate their local server
   set with unreachable addresses.
   
   Cluster administrators can work around this problem by using `server_join` with
   specific IP addresses (or DNS names), because the `Node.UpdateStatus` endpoint
   returns the correct set of RPC addresses when updating the node. So once a
   client has registered, it will get the correct set of RPC addresses.
   
   This changeset updates the client logic to query `Status.Members` instead of
   `Status.Peers`, and then extract the correctly advertised address and port from
   the response body.

   

   tgross committed Feb 28, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 0bf6e65
   • Browse repository at this point

   Copy the full SHA

   0bf6e65 View commit details

   Browse the repository at this point in the history