-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stats API in nodes does not work in private clusters #1717
Comments
Our team has run into this issue as well. We operate a federated cluster in two regions (let's call them A and B). All the Nomad agents advertise their HTTP API on a private IP. We run an instance of hashi-ui in region B. The hashi-ui page which displays allocations' stats and files doesn't work for allocations in region A, because the Nomad API client tries to connect to the client node directly, which of course it cannot do across regions using the client node's region-private IP address. This leaves us in an unpleasant place where we have to run an instance of hashi-ui in each region (which will be confusing to our users, because each instance shows some information about data in the other region, and it won't be clear to most people when they need to switch to the other instance). It might also possibly be solved on our end by using a layer of indirection and having a proxy that forwards to one of the two hashi-ui instances depending on which region is being viewed, but that is a non-trivial workaround too. The proper solution is what @kaskavalci suggests: HTTP API requests should always be made to the Nomad server, and the Nomad server should be responsible for getting the requested data from specific Nomad clients via RPC, federating with other Nomad servers as needed. Since there hasn't been activity on this since October 2016, it doesn't seem like a priority for the Nomad team, so I wanted to add our experience to build further support for improving this. |
#3892 looks like it might solve this problem. Will have to try out in the 0.8 release! |
Done in #3892 in the 0.8 release |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
v0.4.0
Operating system and Environment details
All operating systems
Issue
Stats function in Nodes API does not work in clusters where nomad communicates with private network interfaces. Stats endpoint uses Node.HTTPAddress however that address corresponds to local cluster IP, not an FQDN. Relevant code piece: https://github.com/hashicorp/nomad/blob/master/api/nodes.go#L86
Instead of connecting each node from client side, it would be better if we can query them via server and server will make the connection.
Reproduction steps
Setup nomad cluster with their local IP addresses, not FQDNs. Try to connect stats endpoint from outside.
The text was updated successfully, but these errors were encountered: