api/exec may return "unexpected EOF"

[notnoop]

Our private nightly e2e indicate that the exec API returns "unexpected EOF". This should signal that the exec api client, has gotten an EOF before it receives the exit code. These failures are user visible and disproportionately impact scripted usage of nomad alloc exec.

In a sample of 345 e2e builds, 10 have failed and surfaced this error. The tests in these instances used the api/ package directly, but there were more cases where the nomad alloc exec fails for the same reason but get reported as opaque "status code: ...".

Relevant build output

build-220513.log
1578-[00:24:05] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest
1579-[00:24:05] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEBSVolumeClaim
1580-[00:25:02] :	 [Step 5/6]     csi.go:142:
1581-[00:25:02] :	 [Step 5/6]         	Error Trace:	csi.go:142
1582-[00:25:02] :	 [Step 5/6]         	Error:      	Received unexpected error:
1583:[00:25:02] :	 [Step 5/6]         	            	unexpected EOF

build-219226.log
1535-[05:15:13] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest
1536-[05:15:13] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEBSVolumeClaim
1537-[05:15:53] :	 [Step 5/6]     csi.go:114:
1538-[05:15:53] :	 [Step 5/6]         	Error Trace:	csi.go:114
1539-[05:15:53] :	 [Step 5/6]         	Error:      	Received unexpected error:
1540:[05:15:53] :	 [Step 5/6]         	            	unexpected EOF

build-222935.log
2406-[00:35:01] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/notty:_children_processes
2407-[00:35:05] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_basic
2408-[00:35:07] :	 [Step 5/6]     exec.go:93:
2409-[00:35:07] :	 [Step 5/6]         	Error Trace:	exec.go:93
2410-[00:35:07] :	 [Step 5/6]         	Error:      	Received unexpected error:
2411:[00:35:07] :	 [Step 5/6]         	            	unexpected EOF

build-218710.log
1571-[18:59:52] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEBSVolumeClaim
1572-[19:00:48] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEFSVolumeClaim
1573-[19:01:23] :	 [Step 5/6]     csi.go:196:
1574-[19:01:23] :	 [Step 5/6]         	Error Trace:	csi.go:196
1575-[19:01:23] :	 [Step 5/6]         	Error:      	Received unexpected error:
1576:[19:01:23] :	 [Step 5/6]         	            	unexpected EOF

build-222894.log
2418-[00:33:48] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/notty:_stdin_passing
2419-[00:33:49] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/notty:_children_processes
2420-[00:33:53] :	 [Step 5/6]     exec.go:93:
2421-[00:33:53] :	 [Step 5/6]         	Error Trace:	exec.go:93
2422-[00:33:53] :	 [Step 5/6]         	Error:      	Received unexpected error:
2423:[00:33:53] :	 [Step 5/6]         	            	unexpected EOF

build-220604.log
1631-[00:33:14] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_stdin_passing
1632-[00:33:15] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_children_processes
1633-[00:33:18] :	 [Step 5/6]     exec.go:93:
1634-[00:33:18] :	 [Step 5/6]         	Error Trace:	exec.go:93
1635-[00:33:18] :	 [Step 5/6]         	Error:      	Received unexpected error:
1636:[00:33:18] :	 [Step 5/6]         	            	unexpected EOF

build-222019.log
2220-[02:38:59] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_stdin_passing
2221-[02:39:01] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_children_processes
2222-[02:39:03] :	 [Step 5/6]     exec.go:93:
2223-[02:39:03] :	 [Step 5/6]         	Error Trace:	exec.go:93
2224-[02:39:03] :	 [Step 5/6]         	Error:      	Received unexpected error:
2225:[02:39:03] :	 [Step 5/6]         	            	unexpected EOF

build-218191.log
1568-[01:22:36] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEBSVolumeClaim
1569-[01:23:33] :	 [Step 5/6] === RUN   TestE2E/CSI/*csi.CSIVolumesTest/TestEFSVolumeClaim
1570-[01:24:09] :	 [Step 5/6]     csi.go:196:
1571-[01:24:09] :	 [Step 5/6]         	Error Trace:	csi.go:196
1572-[01:24:09] :	 [Step 5/6]         	Error:      	Received unexpected error:
1573:[01:24:09] :	 [Step 5/6]         	            	unexpected EOF

build-218351.log
1597-[01:26:38] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_stty_check
1598-[01:26:41] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/tty:_stdin_passing
1599-[01:26:42] :	 [Step 5/6]     exec.go:93:
1600-[01:26:42] :	 [Step 5/6]         	Error Trace:	exec.go:93
1601-[01:26:42] :	 [Step 5/6]         	Error:      	Received unexpected error:
1602:[01:26:42] :	 [Step 5/6]         	            	unexpected EOF

build-219979.log
1591-[00:28:37] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/notty:_basic
1592-[00:28:38] :	 [Step 5/6] === RUN   TestE2E/Nomad_exec/*nomadexec.NomadExecE2ETest_(docker)/TestExecBasicResponses/notty:_streaming
1593-[00:28:43] :	 [Step 5/6]     exec.go:93:
1594-[00:28:43] :	 [Step 5/6]         	Error Trace:	exec.go:93
1595-[00:28:43] :	 [Step 5/6]         	Error:      	Received unexpected error:
1596:[00:28:43] :	 [Step 5/6]         	            	unexpected EOF

[massyah]

Our use cases actively depends on scripted usage of nomad alloc exec, triggering approximately ~ 10 execution per minutes; out of these ~ 2% are falsely marked as failure due to this "Unexpected EOF" error.
I'll be happy to provide more details or test a specific build in our setup if needed

[davidtaylorhq]

We have also been seeing frequent occurrences of this "unexpected EOF" error when using nomad alloc exec. Happy to help with collecting more details if needed.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.