Stats API in nodes does not work in private clusters

[kaskavalci]

Nomad version

v0.4.0

Operating system and Environment details

All operating systems

Issue

Stats function in Nodes API does not work in clusters where nomad communicates with private network interfaces. Stats endpoint uses Node.HTTPAddress however that address corresponds to local cluster IP, not an FQDN. Relevant code piece: https://github.com/hashicorp/nomad/blob/master/api/nodes.go#L86

Instead of connecting each node from client side, it would be better if we can query them via server and server will make the connection.

Reproduction steps

Setup nomad cluster with their local IP addresses, not FQDNs. Try to connect stats endpoint from outside.

[jimmyduty]

Our team has run into this issue as well. We operate a federated cluster in two regions (let's call them A and B). All the Nomad agents advertise their HTTP API on a private IP. We run an instance of hashi-ui in region B. The hashi-ui page which displays allocations' stats and files doesn't work for allocations in region A, because the Nomad API client tries to connect to the client node directly, which of course it cannot do across regions using the client node's region-private IP address.

This leaves us in an unpleasant place where we have to run an instance of hashi-ui in each region (which will be confusing to our users, because each instance shows some information about data in the other region, and it won't be clear to most people when they need to switch to the other instance). It might also possibly be solved on our end by using a layer of indirection and having a proxy that forwards to one of the two hashi-ui instances depending on which region is being viewed, but that is a non-trivial workaround too.

The proper solution is what @kaskavalci suggests: HTTP API requests should always be made to the Nomad server, and the Nomad server should be responsible for getting the requested data from specific Nomad clients via RPC, federating with other Nomad servers as needed.

Since there hasn't been activity on this since October 2016, it doesn't seem like a priority for the Nomad team, so I wanted to add our experience to build further support for improving this.

[jimmyduty]

#3892 looks like it might solve this problem. Will have to try out in the 0.8 release!

[preetapan]

Done in #3892 in the 0.8 release

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.