-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad API "Unexpected EOF" Errors #12273
Comments
Hi @evandam I'm sorry you are having issues. I'll take a look a the log and see if anything stands out. I'm curious about something. When this happens have you ever looked at the servers/clients to see how they are doing at the OS level? Specifically, I am wondering if they are experiencing any memory, CPU, or network pressure. |
Also, do you have any client logs showing the error, or are you only experiencing it from the command line? |
Hey @DerekStrickland, adding a couple screenshots of our monitoring - doesn't look like anything unusual in terms of system activity. There was a small spike from ~20% to ~45% CPU, but I believe that was a minute or so after the first EOF error so probably not related. Memory and network pressure seems fine too. There aren't any client logs, we're really just seeing it on the command line/curl. For what it's worth, I've seen this happening on our Nomad client nodes but never on a server (could be coincidence, could be related to different system loads, etc.) but maybe worth pointing out. |
Not sure if it helps, but here's logs after seeing this issue and runnning https://gist.github.com/evandam/69e7dbfa467932831c5cc3460fbe5a79 It helped in a vaguely-similar issue in hashicorp/nomad-autoscaler#514 (comment). |
Sounds good! Just let me know if there's any more info I can provide that would be useful. |
Thanks for the offer. I think at this point I just need to find time to spin up a reproduction test. The working theory is that it can be cause by either a rapid succession of blocking query requests or large number of blocking queries at the same time. If you have a fast way to repro that situation that you can share that would be awesome 😄 Otherwise, it's just on me to get that in the work queue. 🤞 that the theory holds. |
No solution yet, but I did want to share that I have a reproducer. I can't promise it is 100% the same as your issue, but the symptom at least seems the same. |
We experienced same issue again on another node and this time
Here's SIGABRT output https://gist.github.com/ahmedwonolo/63ffe1654d2be5024a0accbf25aa08a9 |
Not sure if this adds any value but I'm seeing the same thing with requests from the nomad autoscaler in #519 also. It's a mix of EOF and connection reset by peer: (I've removed the ip addresses and node ids)
or
Interestingly I get the little ® character in all of my logs as well although I suspect it's meant to be Seems related to #8718 |
Thanks for the added information @spegoraro! |
I can confirm to seeing this behaviour as well quite often. @DerekStrickland anything come up from any investigation thus far? |
@gbolo I'm just wrapping up the work on the nomad Disconnected Clients feature for 1.3. I'm hoping by late next week I'll be freed up to start looking at backlog issues and I've got this at the top of my list of things I'd like to investigate further. I can recreate it easily, so that's a good start. I apologize for the delay, but will definitely update the issue as I progress or if I have any questions. |
Hi @DerekStrickland can you share with us how you are reproducing this exactly? Thanks for looking into it |
Hi @gbolo . For sure. Here's what I did Using the k6 tool I can create an
import { check } from "k6";
import http from "k6/http";
export default function() {
let res = http.get("http://192.168.56.11:4646/v1/jobs?index=23&wait=5m");
check(res, {
"is status 200": (r) => r.status === 200
});
};
k6 run -u 200 -d 20s eof.js
WARN[0005] Request Failed error="Get \"[http://192.168.56.11:4646/v1/jobs?index=23&wait=5m\](http://192.168.56.11:4646/v1/jobs?index=23&wait=5m%5C)": EOF" |
Hello, I was also able to reproduce the error running Nomad locally and using k6 tool:
|
Nomad version
Nomad v1.2.6 (a6c6b475db5073e33885377b4a5c733e1161020c)
I've noticed this issue going back a few versions, unsure of when it began but possibly < 1.0.0.
Operating system and Environment details
Ubuntu 18.04
Issue
We sporadically see API calls on Nomad clients start to fail with "Unexpected EOF" errors.
Reproduction steps
Unclear, seemingly random on Nomad clients.
Expected Result
Nomad API handles requests correctly
Actual Result
$ nomad server members Error querying servers: Get "http://127.0.0.1:4646/v1/agent/members": EOF $ curl localhost:4646/v1/status/leader curl: (56) Recv failure: Connection reset by peer
Job file (if appropriate)
N/A
Nomad Server logs (if appropriate)
There's nothing that jumps out in server logs, but here was the last hour before an instance of this error. Note that the "permission denied" errors can be safely ignored, it's just checks we have to verify ACL permissions in place.
https://gist.github.com/evandam/1d1f2dd2a5032426736107f414959e2d
Nomad Client logs (if appropriate)
N/A
I know it's not much to go on, but let me know if there's any additional detail I can provide. I'll typically see this occurring on a host until I restart the Nomad service.
The text was updated successfully, but these errors were encountered: