-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No stats in alloc-status in nomad 0.8.1 (no path to node) #4203
Comments
@commarla Is that node that was running that allocation still alive? Have all nodes and servers been upgraded? Do you see this for all node-status calls? |
@dadgar Yes the node is still alive. I can reproduce on any 0.8.1 nodes. We have some nodes in 0.7.x with a different class. (20 nodes in 0.8.1, 20 nodes in 0.7.1 and 10 in 0.7.0). On a node in 0.7 I have a 500 :
|
@commarla Do you have any reproduction steps? I can not reproduce it. |
@dadgar I have this error all the time, I don't know what to do to help you reproduce. The cluster was updated from 0.7.1 and before that 0.6.3, 0.7.0-beta1, 0.7.1-rc1 |
@nanoz do you have something to add ? |
Even though I can GET allocs and nodes informations, stats are the only endpoints not working properly, probably because of the RPC proxying feature. Allocation API calls
Node API calls
The HTTP call works fine if we curl it from the nomad server to the client node.
My understanding is that Nomad 0.8 now proxies those calls using the RPC port. I think the problem is that we are not listening on anything else than 4646 on the client node.
Here is our client node configuration
Isn't bind_addr enough to activate RPC? I also wonder why are we obligated to go through the server RPC proxying, isn't the "old way" of retrieving stats by requesting the client's HTTP API directly the default way of retrieving stats? |
A few questions:
|
thanks @dadgar , we will test this in the next two weeks |
Nomad v0.8.3 (c85483d) New installation.
Error: Couldn't retrieve stats: Unexpected response code: 404 (No path to node) Logs from the node on which the command is run:
Logs from the remote node on which the container is running:
If you run the command
|
@grin0c i confirm, same here. |
@yellowmegaman I confirm |
@grin0c @yellowmegaman Any steps to follow to reproduce? The log message I see how to reproduce (it is a noisy log that I will fix) but more asking about the 404. Can you all share your client/server configs/how long it takes for this to happen, what you do to the nodes/servers/etc for this to occur. |
@dadgar this is my config:
On other nodes, only the ip is changed What I do for this: |
@grin0c Are you using Consul as well? If you are can you share: |
|
@grin0c Could you also display |
|
@yellowmegaman Are you also running servers and clients together? |
@dadgar yes, i can provide addotional info tomorrow. We're using small clusters, 5-7 nodes, sometimes on baremetal, so all nodes are clients too, since running server-only mode on 3 nodes would be serious waste of resources. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.8.1 (46aa11b) on both server and client
Operating system and Environment details
Debian Jessie.
Cluster was updated from 0.7.1
Issue
Got a
Couldn't retrieve stats: Unexpected response code: 404 (No path to node)
duringnomad alloc-status
command.Got an error with
nomad node-status
:error fetching node stats: actual resource usage not present
Reproduction steps
Start a job, try to get the alloc-status.
Nomad Server logs (if appropriate)
The text was updated successfully, but these errors were encountered: