client: use RPC address and not serf after initial Consul discovery #16217

tgross · 2023-02-17T20:21:52Z

Nomad servers can advertise independent IP addresses for serf and rpc. Somewhat unexpectedly, the serf address is also used for both Serf and server-to-server RPC communication (including Raft RPC). The address advertised for rpc is only used for client-to-server RPC. This split was introduced intentionally in Nomad 0.8.

When clients are using Consul discovery for connecting to servers, they get an initial discovery set from Consul and use the correct rpc tag in Consul to get a list of adddresses for servers. The client then makes a Status.Peers RPC to get the list of those servers that are raft peers. But this endpoint is shared between servers and clients, and provides the address used for Raft.

Most of the time this is harmless because servers will bind on 0.0.0.0 anyways., But in topologies where servers are on a private network and clients are on separate subnets (or even public subnets), clients will make initial contact with the server to get the list of peers but then populate their local server set with unreachable addresses.

Cluster administrators can work around this problem by using server_join with specific IP addresses (or DNS names), because the Node.UpdateStatus endpoint returns the correct set of RPC addresses when updating the node. So once a client has registered, it will get the correct set of RPC addresses.

This changeset updates the client logic to query Status.Members instead of Status.Peers, and then extract the correctly advertised address and port from the response body.

Notes:

The end-to-end testing required for this means it's going to just miss the 1.5.0 GA. This is probably safe to land in a patch release without worrying about backwards compatibility, because anyone with this topology is already using the server_join workaround described above.
This also closes fix wrong control plane discovery, when serf advertised address located in network, unreachable from clients #11895, which unfortunately got abandoned because we had some trouble understanding exactly what the problem was.
I've done some bench testing of this with a local development cluster, some extra IP addresses, and iptables rules.
I also stood it up on a real multi-host cluster using the changes in our E2E config from E2E: add multi-home networking to test infrastructure #16218 and everything seems to work as usual.
This is outside the code path of the non-Consul cloud discovery features, but I also tested it by disabling server join via Consul (by just breaking the consul config block) and used AWS EC2 auto-join via tags and that works just fine as well.

Add an Elastic Network Interface (ENI) to each Linux host, on a secondary subnet we have provisioned in each AZ. Revise security groups as follows: * Split out client security groups from servers so that we can't have clients accidentally accessing serf addresses or other unexpected cross-talk. * Add new security groups for the secondary subnet that only allows communication within the security group so we can exercise behaviors with multiple IPs. This changeset doesn't include any Nomad configuration changes needed to take advantage of the extra network interface. I'll include those with testing for PR #16217.

Nomad servers can advertise independent IP addresses for `serf` and `rpc`. Somewhat unexpectedly, the `serf` address is also used for both Serf and server-to-server RPC communication (including Raft RPC). The address advertised for `rpc` is only used for client-to-server RPC. This split was introduced intentionally in Nomad 0.8. When clients are using Consul discovery for connecting to servers, they get an initial discovery set from Consul and use the correct `rpc` tag in Consul to get a list of adddresses for servers. The client then makes a `Status.Peers` RPC to get the list of those servers that are raft peers. But this endpoint is shared between servers and clients, and provides the address used for Raft. Most of the time this is harmless because servers will bind on 0.0.0.0 anyways., But in topologies where servers are on a private network and clients are on separate subnets (or even public subnets), clients will make initial contact with the server to get the list of peers but then populate their local server set with unreachable addresses. Cluster administrators can work around this problem by using `server_join` with specific IP addresses (or DNS names), because the `Node.UpdateStatus` endpoint returns the correct set of RPC addresses when updating the node. So once a client has registered, it will get the correct set of RPC addresses. This changeset updates the client logic to query `Status.Members` instead of `Status.Peers`, and then extract the correctly advertised address and port from the response body.

lgfa29

LGTM!

…16217) Nomad servers can advertise independent IP addresses for `serf` and `rpc`. Somewhat unexpectedly, the `serf` address is also used for both Serf and server-to-server RPC communication (including Raft RPC). The address advertised for `rpc` is only used for client-to-server RPC. This split was introduced intentionally in Nomad 0.8. When clients are using Consul discovery for connecting to servers, they get an initial discovery set from Consul and use the correct `rpc` tag in Consul to get a list of adddresses for servers. The client then makes a `Status.Peers` RPC to get the list of those servers that are raft peers. But this endpoint is shared between servers and clients, and provides the address used for Raft. Most of the time this is harmless because servers will bind on 0.0.0.0 anyways., But in topologies where servers are on a private network and clients are on separate subnets (or even public subnets), clients will make initial contact with the server to get the list of peers but then populate their local server set with unreachable addresses. Cluster administrators can work around this problem by using `server_join` with specific IP addresses (or DNS names), because the `Node.UpdateStatus` endpoint returns the correct set of RPC addresses when updating the node. So once a client has registered, it will get the correct set of RPC addresses. This changeset updates the client logic to query `Status.Members` instead of `Status.Peers`, and then extract the correctly advertised address and port from the response body.

In #16217 we switched clients using Consul discovery to the `Status.Members` endpoint for getting the list of servers so that we're using the correct address. This endpoint is authenticated, and we failed to add the `AuthToken` with the client secret to the request. This changeset also expands the test coverage of the RPC endpoint and closes up potential holes in the `ResolveACL` method that aren't currently bugs but easily could become bugs if we called the method without ensuring its invariants are upheld.

In #16217 we switched clients using Consul discovery to the `Status.Members` endpoint for getting the list of servers so that we're using the correct address. This endpoint has an authorization gate, so this fails if the anonymous policy doesn't have `node:read`. We also can't check the `AuthToken` for the request for the client secret, because the client hasn't yet registered so the server doesn't have anything to compare against. Create a new `Status.RPCServers` endpoint that mirrors the `Status.Peers` endpoint but provides the RPC server addresses instead of the Raft addresses. This fixes the authentication bug but also ensures we're only registering with servers in the client's region and not in any other servers that might have registered with Consul. This changeset also expands the test coverage of the RPC endpoint and closes up potential holes in the `ResolveACL` method that aren't currently bugs but easily could become bugs if we called the method without ensuring its invariants are upheld.

In #16217 we switched clients using Consul discovery to the `Status.Members` endpoint for getting the list of servers so that we're using the correct address. This endpoint has an authorization gate, so this fails if the anonymous policy doesn't have `node:read`. We also can't check the `AuthToken` for the request for the client secret, because the client hasn't yet registered so the server doesn't have anything to compare against. Create a new `Status.RPCServers` endpoint that mirrors the `Status.Peers` endpoint but provides the RPC server addresses instead of the Raft addresses. This fixes the authentication bug but also ensures we're only registering with servers in the client's region and not in any other servers that might have registered with Consul. This changeset also expands the test coverage of the RPC endpoint and closes up potential holes in the `ResolveACL` method that aren't currently bugs but easily could become bugs if we called the method without ensuring its invariants are upheld. Co-authored-by: tantra35 <ruslan.usifov@gmail.com>

In #16217 we switched clients using Consul discovery to the `Status.Members` endpoint for getting the list of servers so that we're using the correct address. This endpoint has an authorization gate, so this fails if the anonymous policy doesn't have `node:read`. We also can't check the `AuthToken` for the request for the client secret, because the client hasn't yet registered so the server doesn't have anything to compare against. Instead of hitting the `Status.Peers` or `Status.Members` RPC endpoint, use the Consul response directly. Update the `registerNode` method to handle the list of servers we get back in the response; if we get a "no servers" or "no path to region" response we'll kick off discovery again and retry immediately rather than waiting 15s.

In #16217 we switched clients using Consul discovery to the `Status.Members` endpoint for getting the list of servers so that we're using the correct address. This endpoint has an authorization gate, so this fails if the anonymous policy doesn't have `node:read`. We also can't check the `AuthToken` for the request for the client secret, because the client hasn't yet registered so the server doesn't have anything to compare against. Instead of hitting the `Status.Peers` or `Status.Members` RPC endpoint, use the Consul response directly. Update the `registerNode` method to handle the list of servers we get back in the response; if we get a "no servers" or "no path to region" response we'll kick off discovery again and retry immediately rather than waiting 15s. Co-authored-by: Tim Gross <tgross@hashicorp.com>

vercel bot deployed to Preview – nomad-storybook-and-ui February 17, 2023 20:27 View deployment

tgross added this to the 1.5.x milestone Feb 17, 2023

tgross added theme/client type/bug labels Feb 17, 2023

tgross force-pushed the client-discover-by-members-rpc branch from c1b0c79 to 3c094d3 Compare February 17, 2023 20:32

This was referenced Feb 17, 2023

fix wrong control plane discovery, when serf advertised address located in network, unreachable from clients #11895

Closed

Nomad client tries to connect to serf IP address, not to RPC #16211

Closed

vercel bot deployed to Preview – nomad-storybook-and-ui February 17, 2023 20:39 View deployment

tgross self-assigned this Feb 17, 2023

tgross mentioned this pull request Feb 17, 2023

E2E: add multi-home networking to test infrastructure #16218

Merged

tgross marked this pull request as ready for review February 28, 2023 21:53

tgross requested review from lgfa29 and angrycub February 28, 2023 21:53

tgross added backport/1.3.x backport to 1.3.x release line backport/1.4.x backport to 1.4.x release line labels Feb 28, 2023

tgross force-pushed the client-discover-by-members-rpc branch from 3c094d3 to 0bf6e65 Compare February 28, 2023 21:55

vercel bot deployed to Preview – nomad-storybook-and-ui February 28, 2023 22:01 View deployment

lgfa29 approved these changes Mar 1, 2023

View reviewed changes

tgross added the backport/1.5.x backport to 1.5.x release line label Mar 2, 2023

tgross merged commit a9bb8e9 into main Mar 2, 2023

tgross deleted the client-discover-by-members-rpc branch March 2, 2023 18:36

tgross modified the milestones: 1.5.x, 1.5.1 Mar 6, 2023

tgross mentioned this pull request Mar 13, 2023

clients require node:read on anon ACL policy to join cluster via Consul discovery (1.5.1) #16470

Closed

tgross mentioned this pull request Mar 14, 2023

client: don't use Status RPC for Consul discovery (#16490) #16490

Merged

tgross mentioned this pull request Mar 15, 2023

E2E: advertise serf on secondary IP #16507

Open

2 tasks

tgross mentioned this pull request Mar 21, 2023

Nomad 1.5.1 client unable to start with IPv6 server addresses #16586

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: use RPC address and not serf after initial Consul discovery #16217

client: use RPC address and not serf after initial Consul discovery #16217

tgross commented Feb 17, 2023 •

edited

Loading

lgfa29 left a comment

client: use RPC address and not serf after initial Consul discovery #16217

client: use RPC address and not serf after initial Consul discovery #16217

Conversation

tgross commented Feb 17, 2023 • edited Loading

lgfa29 left a comment

Choose a reason for hiding this comment

tgross commented Feb 17, 2023 •

edited

Loading