-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider using socketpair on Unix for Process.Start redirected stdin/out/err #34158
Comments
The identical effect (with very different code) holds true for Windows, where it eventually uses CreatePipe to get anonymous pipe pairs, which do not support overlapped operations. Here: runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Windows.cs Line 828 in a54d391
Could that be changed to a pair of named pipes for example, which would allow the full power of overlapped/IOCP? I guess it would merit its own issue because the two pieces of code are unrelated. |
Maybe, but see the large comment just below that. |
Ah yes... it would be a breaking change to processes that eventually WriteFile without an overlapped pointer. Could this be opt-in for cases where the subprocess is known by the caller to be compatible? But the identical synchronous behavior could be kept if the subprocess-side of the named pipe is non-overlapped, no? (Sorry for kind of hijacking the linux/mac topic!) |
It would be great if we improve this. Next to potential ThreadPool exhaustion, there is also no way to abort the pending read that is blocking the ThreadPool thread. I think using
We should check how read/write behave when the .NET end has closed, and verify if it is similar to pipes. I expect this to be the case.
fyi, for the |
Doing this correctly will likely require #862. |
@davidfowl, could you try this out and see if it addresses the issue you hit in: Here's a branch: Perf of sync operations on the Process.StandardOutput/Input/Error is about the same. There's more CPU overhead now when using the async operations than there was before, by a non-trivial amount, but that's also pretty common: there's more set up to be done, more syscalls to be made when data isn't available, etc. However, the async operations also now allocate significantly less than they did before, don't block threads, and are cancelable. On the balance, I think it's a reasonable change to make, assuming it unblocks the scenarios we care about. @tmds, @wfurt, obviously please feel free to take it for a spin as well. I'm not putting up a PR yet, as getting it in is blocking on approving #862. |
How much of the clr do I have to build to get this. |
System.Net.Sockets.dll, System.Diagnostics.Process.dll, and libSystem.Native.so. You could build them individually if you want, but it's probably easier just to use the command I shared to build everything. Up to you :) |
On Linux and macOS today when we fork a process as part of Process.Start, if RedirectStandardInput/Output/Error are set, we create anonymous pipes and use those for the new process' stdin/out/err. The resulting file descriptors are then wrapped in FileStreams, which are in turn wrapped in StreamReader/Writers.
A downside of this, however, is that async operations performed on those readers/writers end up doing async-over-sync as part of the FileStream implementation, which just queues work items that do Read/Write as part of ReadAsync/WriteAsync, and that can be problematic for systems that spawn lots of processes and monitor their stdout asynchronously.
There are a few ways to address that performance issue. One is to enable the epoll-based architecture we have underlying System.Net.Sockets to be used by any type with any relevant kind of file descriptor, but that is a significant effort. In the meantime, we might be able to achieve just as good a result, maybe better, specifically for Process, by using socketpair instead of pipe, and constructing NetworkStreams with Sockets instead of FileStreams. This would end up implicitly using the right support under sockets, it would end up providing more efficient implementations of APIs like ReadAsync (which has a non-allocating implementation in NetworkStream), etc.
@tmds, @wfurt, any significant roadblocks to this spring to mind?
The text was updated successfully, but these errors were encountered: