asyncio subprocess stdin/out/err can be filehandles, but this is undocumented

[bewinsnw]

Documentation

I was trying to write some asyncio subprocess code to pipe from one process to another, struggled until I found examples with os.pipe(). But going back to the docs I was confused because passing a filehandle isn't mentioned as an option. Looking at the underlying code, I see it definitely is supported, there's a check for the parameter being an integer and the handle is used directly.

The incorrect documentation is here

cpython/Doc/library/asyncio-eventloop.rst

Lines 1439 to 1469 in 101d5ec

	* *stdin* can be any of these:
	* a file-like object
	* the :const:`subprocess.PIPE` constant (default) which will create a new
	pipe and connect it,
	* the value ``None`` which will make the subprocess inherit the file
	descriptor from this process
	* the :const:`subprocess.DEVNULL` constant which indicates that the
	special :data:`os.devnull` file will be used
	* *stdout* can be any of these:
	* a file-like object
	* the :const:`subprocess.PIPE` constant (default) which will create a new
	pipe and connect it,
	* the value ``None`` which will make the subprocess inherit the file
	descriptor from this process
	* the :const:`subprocess.DEVNULL` constant which indicates that the
	special :data:`os.devnull` file will be used
	* *stderr* can be any of these:
	* a file-like object
	* the :const:`subprocess.PIPE` constant (default) which will create a new
	pipe and connect it,
	* the value ``None`` which will make the subprocess inherit the file
	descriptor from this process
	* the :const:`subprocess.DEVNULL` constant which indicates that the
	special :data:`os.devnull` file will be used
	* the :const:`subprocess.STDOUT` constant which will connect the standard
	error stream to the process' standard output stream

So instead of

   * *stdin* can be any of these:

     * a file-like object
     * the :const:`subprocess.PIPE` constant (default) which will create a new
       pipe and connect it,...

I'd suggest:

   * *stdin* can be any of these:
  
     * a file-like object
     * an existing file descriptor (a positive integer), for example those created with :meth:`os.pipe()`,
     * the :const:`subprocess.PIPE` constant (default) which will create a new
       pipe and connect it,...

(and similarly for stdout and stderr)

Linked PRs

• gh-105857: Document that asyncio subprocess std{in,out,err} can be file handles #107986
• [3.12] gh-105857: Document that asyncio subprocess std{in,out,err} can be file handles (GH-107986) #108332

[gvanrossum]

I'm not sure if this is just a lack of documentation or whether it is intentional, to discourage using raw file descriptors? There might be some problems in edge cases (e.g. sockets on Windows). If you have a simple example of how you used this successfully that would also help.

[bewinsnw]

example:

import asyncio
import os

async def test():
    r,w = os.pipe()
    p1 = asyncio.create_subprocess_shell("seq 1 100000", stdout=w)
    p2 = asyncio.create_subprocess_shell("wc", stdin=r)
    await asyncio.gather(p1, p2)

asyncio.run(test())

$ python3 spike.py
  100000  100000  588895                                                                                                                                                                     

(the actual code I'm using is a fair bit more complex and takes care of closing handles etc).

I think it's just an omission, not intentional. The asyncio subprocess functions are mostly just wrappers for Popen, which is the thing handling the file handles; and it's handled in both unix and windows variants.

The non-asyncio subprocess documents that file descriptors are ok: https://docs.python.org/3/library/subprocess.html#frequently-used-arguments "stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object with a valid file descriptor, and None."

In the asyncio subprocess implementation, _UnixSubprocessTransport passes everything but subprocess.PIPE to Popen:

cpython/Lib/asyncio/unix_events.py

Lines 821 to 823 in 1858db7

	self._proc = subprocess.Popen(
	args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr,
	universal_newlines=False, bufsize=bufsize, **kwargs)

Popen calls _get_handles:

cpython/Lib/subprocess.py

Line 992 in 1858db7

errread, errwrite) = self._get_handles(stdin, stdout, stderr)

which handles the int filehandles case: https://github.com/python/cpython/blob/main/Lib/subprocess.py#L1704-L1705

_WindowsSubprocessTransport is even simpler and passes all of the stream arguments to Popen: https://github.com/python/cpython/blob/1858db7cbdbf41aa600c954c15224307bf81a258/Lib/asyncio/windows_events.py#L873C1-L875

[gvanrossum]

Okay, makes sense. Would you like to submit a doc PR? (I wouldn't mind if the docs were refactored to avoid repeating everything three times, though maybe there are differences that make this tricky, so up to you.)

[Hels15]

I have added a pr here, extending the docs. #107986