Spurious error messages when tasks exit successfully

[schmichael]

When running a batch job that exits successfully, RPC error messages are still emitted in the client agent logs:

[ERROR] client.driver_mgr.exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=eb3a3678-9053-198a-118e-0b4fbf61256a driver=exec task_name=echo error="rpc error: code = Unavailable desc = transport is closing"

Nomad Version

Most recently 1.1.2 and 0.12.11, but similar reports go back to 0.10 and 0.9.

References

• Same spurious error but it also failed the job: client.driver_mgr.exec: error receiving stream from Stats executor RPC #7061
• Same spurious error: using exec driver result into RPC error #5986

Jobspec

Should be able to reproduce with any batch job that exits successfully:

job "echo" {
	datacenters = ["dc1"]
	type = "batch"

	group "cache" {
		task "echo" {
			driver = "exec"
			config {
  				command = "/bin/sleep"
    				args    = ["1"]
			}
		}
	}
}

[anastazya]

I can confirm the same messages when running lots of RawExec jobs.

[jseba]

We're hitting this as well with our plugin, using Nomad version 1.0.4. From our logs, we see this message over 7000 times in the last 12 hours (as of this message), all of them with the same error code, "task not found for given id" (ErrTaskNotFound). However, not all of the jobs are type = "batch", some of them are service jobs.

[T0tt1]

Do we have any progress here? I can report the same behaviour and searching for solution. Nomad v1.0.4

/bin/bash mkdir <my_desired_folder>

@message | {"@level":"error","@message":"error receiving stream from Stats executor RPC, closing stream","@module":"client.driver_mgr.raw_exec","@timestamp":"2021-09-17T04:06:37.101555Z","alloc_id":"#######-####-####-####-############","driver":"raw_exec","error":"rpc error: code = Unavailable desc = transport is closing","task_name":"qaz-wsx-edc"}

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.