-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spurious error messages when tasks exit successfully #10814
Comments
I can confirm the same messages when running lots of RawExec jobs. |
We're hitting this as well with our plugin, using Nomad version 1.0.4. From our logs, we see this message over 7000 times in the last 12 hours (as of this message), all of them with the same error code, "task not found for given id" (ErrTaskNotFound). However, not all of the jobs are |
Do we have any progress here? I can report the same behaviour and searching for solution. Nomad v1.0.4
|
Suppress stats streaming error log messages when task finishes. Streaming errors are expected when a task finishes and they aren't actionable to users. Also, note that the task runner Stats hook retries collecting stats after a delay. If the connection terminates prematurely, it will be retried, and closing the stats stream is not very disruptive. Ideally, executor terminates cleanly when task exits, but that's a more substantial change that may require changing the executor/drivers interface. Fixes #10814
Suppress stats streaming error log messages when task finishes. Streaming errors are expected when a task finishes and they aren't actionable to users. Also, note that the task runner Stats hook retries collecting stats after a delay. If the connection terminates prematurely, it will be retried, and closing the stats stream is not very disruptive. Ideally, executor terminates cleanly when task exits, but that's a more substantial change that may require changing the executor/drivers interface. Fixes #10814
Suppress stats streaming error log messages when task finishes. Streaming errors are expected when a task finishes and they aren't actionable to users. Also, note that the task runner Stats hook retries collecting stats after a delay. If the connection terminates prematurely, it will be retried, and closing the stats stream is not very disruptive. Ideally, executor terminates cleanly when task exits, but that's a more substantial change that may require changing the executor/drivers interface. Fixes #10814
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
When running a batch job that exits successfully, RPC error messages are still emitted in the client agent logs:
Nomad Version
Most recently 1.1.2 and 0.12.11, but similar reports go back to 0.10 and 0.9.
References
exec
driver result into RPC error #5986Jobspec
Should be able to reproduce with any batch job that exits successfully:
The text was updated successfully, but these errors were encountered: