Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create_subprocess_exec stdin writer does not write everything #312

Closed
tardyp opened this issue Feb 10, 2020 · 2 comments · Fixed by #342
Closed

create_subprocess_exec stdin writer does not write everything #312

tardyp opened this issue Feb 10, 2020 · 2 comments · Fixed by #342

Comments

@tardyp
Copy link
Contributor

tardyp commented Feb 10, 2020

  • Python version:
    Darwin, python 3.7.4
# pipenv install pytest uvloop pytest-aiohttp
# pipenv run pip freeze
aiohttp==3.6.2
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
idna==2.8
importlib-metadata==1.5.0
more-itertools==8.2.0
multidict==4.7.4
packaging==20.1
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
pytest==5.3.5
pytest-aiohttp==0.3.0
six==1.14.0
uvloop==0.14.0
wcwidth==0.1.8
yarl==1.4.2
zipp==2.2.0

Versions
This problem was already detected on my prod on linux.

platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1

  • Platform:
    linux/darwin

  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
    yes

  • Does uvloop behave differently from vanilla asyncio? How?:

yes, test do not hang with pyloop.

When writing large amount of data in the stdin of a subprocess, we observe sometimes that the subprocess will hang trying to read from our app.

Trying to reproduce with a very small test, I find that, on macos, a limit of 8KiB.

Here is my test

# Standard Library
import asyncio.subprocess
import pytest

@pytest.mark.parametrize("num_lines", range(8188, 8194,1))
async def test_uvloopwrite(loop, num_lines):
    proc = await asyncio.create_subprocess_exec("python", __file__,
                                                        stdout=asyncio.subprocess.PIPE,
                                                        stdin=asyncio.subprocess.PIPE)
    data =  b"\n" * num_lines + b"END\n"
    print(num_lines, len(data))
    print("writing", proc.stdin.transport.get_write_buffer_limits())
    proc.stdin.write(data)
    print("drain")
    await proc.stdin.drain()
    # proc.stdin.close()
    print("waiting")
    await proc.wait()
    out = await proc.stdout.read()
    assert int(out) == num_lines

# this small program is called by the test
if __name__ == "__main__":
    import sys
    n = 0
    while True:
        line = sys.stdin.readline()
        if n > 8186:
            print(n, repr(line), file=sys.stderr)
        if line == "END\n":
            break
        n+=1
    print(n)

I run it with :

pipenv install pytest uvloop pytest-aiohttp
pipenv run pytest -s --aiohttp-loop=uvloop test_uvloop.py

When printing on the subprocess, I can see that the first 8191 bytes are seen by the subprocess, but the last one are never seen.

If I close the stdin (line commented in the test), it appears that it forces a buffer flush, and the test finishes.

@1st1
Copy link
Member

1st1 commented Feb 11, 2020

Would appreciate you submitting a PR with a functional test (marked as x-fail) to demonstrate the problem.

@tardyp
Copy link
Contributor Author

tardyp commented Feb 12, 2020

done. actually I re-tested under linux, and found out a weird threshold of 219264 bytes

tardyp added a commit to tardyp/uvloop that referenced this issue Feb 12, 2020
tardyp added a commit to tardyp/uvloop that referenced this issue Feb 12, 2020
fantix added a commit to fantix/uvloop that referenced this issue May 9, 2020
* in order to detect peer close on O_WRONLY pipe_t
* partially reverted d8fe153
* refs libuv/libuv#2058
* refs MagicStack#317
* fixes MagicStack#311, fixes MagicStack#312
fantix pushed a commit to fantix/uvloop that referenced this issue May 9, 2020
fantix pushed a commit to fantix/uvloop that referenced this issue May 9, 2020
fantix added a commit to fantix/uvloop that referenced this issue May 13, 2020
* in order to detect peer close on O_WRONLY pipe_t
* partially reverted d8fe153
* refs libuv/libuv#2058
* refs MagicStack#317
* fixes MagicStack#311, fixes MagicStack#312
fantix pushed a commit to fantix/uvloop that referenced this issue May 13, 2020
fantix added a commit that referenced this issue May 14, 2020
* in order to detect peer close on O_WRONLY pipe_t
* partially reverted d8fe153
* refs libuv/libuv#2058
* refs #317
* fixes #311, fixes #312
fantix pushed a commit that referenced this issue May 14, 2020
This was referenced Feb 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants