Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available #2277

segal90 · 2017-09-18T20:26:57Z

Long story short

Content of a POST response is None or b'' (based on how do you want access the response content) if "Upgrade: h2,h2c" and "Connection: Upgrade" headers are in the HTTP response.
After disable Apache's mod_http2 everything works like a charm.

Expected behaviour

Ignore these headers or handle correctly the HTTP2 upgrade request but not ignore the response body.

Actual behaviour

Response body is ignored

Steps to reproduce

configure an Apache with mod_http2
run this code snippet:

auth = aiohttp.BasicAuth('user1', 'VerySecretPassword')
data = {'just': 'some', 'dummy': 'data'}
async with aiohttp.ClientSession(auth=auth, conn_timeout=10) as session:
    async with session.post(<SERVER_ADDRESS>, data=data) as response:
        print(await response.json())

Your environment

OS: Linux (Fedora)
aiohttp version: 2.2.5
uvloop is set as asyncio event loop

asvetlov · 2017-09-20T20:52:33Z

aiohttp doesn't support HTTP/2 yet.

Please describe what response do you request for the case.

segal90 · 2017-09-20T21:20:50Z

Lack of HTTP/2 is not the problem. Problem is that the HTTP client is not ignore the request for HTTP2 from the webserver.
I just want access the HTTP response body which is now missing.

Findus23 · 2018-01-27T21:20:34Z

Hi,

I am having the same error with https://www.soeren-hentzschel.at/feed/.

Testcase:
https://gist.github.com/Findus23/3aaedbac0dd2520a9b65d0c9a6431b22

asvetlov · 2018-01-27T21:23:30Z

Would you provide a Pull Request with fix?

Findus23 · 2018-01-27T21:24:53Z

Unfortunately I am really a beginner in asynchronous python and just by change came across this error.

asvetlov · 2018-01-30T11:37:10Z

Cannot reproduce on aiohttp master, both Cython and Pure Python parsers.

I've tried a code from https://gist.github.com/Findus23/3aaedbac0dd2520a9b65d0c9a6431b22

Findus23 · 2018-01-30T11:59:44Z

That's really odd.
I tried it again with a clean venv and am I using Python 3.6.4 on Debian Testing. The code is the same as before.
With the master I get aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Exact steps to reproduce

➜  ~ mktmpenv --python=python3 
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/python3
Also creating executable in /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/preactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/postactivate
virtualenvwrapper.user_scripts creating /home/lukas/.virtualenvs/tmp-b68734831c06cd1f/bin/get_env_details
This is a temporary environment. It will be deleted when you run 'deactivate'.
➜  ~ python --version
Python 3.6.4
➜  ~ nano test.py
➜  ~ pip install aiohttp
Collecting aiohttp
  Using cached aiohttp-2.3.9-cp36-cp36m-manylinux1_x86_64.whl
Collecting async-timeout>=1.2.0 (from aiohttp)
  Using cached async_timeout-2.0.0-py3-none-any.whl
Collecting yarl>=1.0.0 (from aiohttp)
  Using cached yarl-1.1.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting multidict>=4.0.0 (from aiohttp)
  Downloading multidict-4.1.0-cp36-cp36m-manylinux1_x86_64.whl (482kB)
    100% |████████████████████████████████| 491kB 1.1MB/s 
Collecting chardet (from aiohttp)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting idna>=2.0 (from yarl>=1.0.0->aiohttp)
  Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: async-timeout, multidict, idna, yarl, chardet, aiohttp
Successfully installed aiohttp-2.3.9 async-timeout-2.0.0 chardet-3.0.4 idna-2.6 multidict-4.1.0 yarl-1.1.0
➜  ~ pip freeze
aiohttp==2.3.9
async-timeout==2.0.0
chardet==3.0.4
idna==2.6
multidict==4.1.0
yarl==1.1.0
➜  ~ python test.py 
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 11:50:58 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>


True
➜  ~ pip install git+https://github.com/aio-libs/aiohttp.git
Collecting git+https://github.com/aio-libs/aiohttp.git
  Cloning https://github.com/aio-libs/aiohttp.git to /tmp/pip-nefx0_40-build
Collecting attrs>=17.4.0 (from aiohttp==3.0.0a0)
  Using cached attrs-17.4.0-py2.py3-none-any.whl
Requirement already satisfied: chardet in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: multidict>=4.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: async_timeout>=1.2.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: yarl>=1.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already satisfied: idna>=2.0 in ./lib/python3.6/site-packages (from yarl>=1.0.0->aiohttp==3.0.0a0)
Installing collected packages: attrs, aiohttp
  Found existing installation: aiohttp 2.3.9
    Uninstalling aiohttp-2.3.9:
      Successfully uninstalled aiohttp-2.3.9
  Running setup.py install for aiohttp ... done
Successfully installed aiohttp-3.0.0a0 attrs-17.4.0
➜  ~ python test.py 
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 11:52:45 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "test.py", line 16, in main
    html = await fetch(session, 'https://www.soeren-hentzschel.at/feed/')
  File "test.py", line 11, in fetch
    return await response.text()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 830, in text
    await self.read()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 799, in read
    self._content = await self.content.read()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 312, in read
    block = await self.readany()
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 328, in readany
    await self._wait('readany')
  File "/home/lukas/.virtualenvs/tmp-b68734831c06cd1f/lib/python3.6/site-packages/aiohttp/streams.py", line 250, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Findus23 · 2018-01-30T12:13:46Z

#2588 seems to fix the issue for me:

➜  ~ pip install git+https://github.com/lphuberdeau/aiohttp.git@limit-upgrade-values -U
Collecting git+https://github.com/lphuberdeau/aiohttp.git@limit-upgrade-values
  Cloning https://github.com/lphuberdeau/aiohttp.git (to limit-upgrade-values) to /tmp/pip-lapcl_xq-build
Requirement already up-to-date: chardet in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: multidict>=3.0.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: async_timeout>=1.2.0 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: yarl>=0.11 in ./lib/python3.6/site-packages (from aiohttp==3.0.0a0)
Requirement already up-to-date: idna>=2.0 in ./lib/python3.6/site-packages (from yarl>=0.11->aiohttp==3.0.0a0)
Installing collected packages: aiohttp
  Found existing installation: aiohttp 3.0.0a0
    Uninstalling aiohttp-3.0.0a0:
      Successfully uninstalled aiohttp-3.0.0a0
  Running setup.py install for aiohttp ... done
Successfully installed aiohttp-3.0.0a0
➜  ~ python test.py                                                                    
<ClientResponse(https://www.soeren-hentzschel.at/feed/) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 30 Jan 2018 12:12:20 GMT', 'Server': 'Apache', 'Last-Modified': 'Mon, 29 Jan 2018 22:41:58 GMT', 'ETag': '"95b04997e00647ea795eb5503aad9780"', 'X-Robots-Tag': 'noindex, follow', 'Link': '<https://www.soeren-hentzschel.at/wp-json/>; rel="https://api.w.org/"', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/rss+xml; charset=UTF-8')>

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
        xmlns:content="http://purl.org/rss/1.0/modules/content/"
        xmlns:wfw="http://wellformedweb.org/CommentAPI/"
        xmlns:dc="http://purl.org/dc/elements/1.1/"
[AND THE REST OF THE RSS FEED]

asvetlov · 2018-01-30T12:22:31Z

Fixed on master.

Findus23 · 2018-01-30T12:25:21Z

Isn't the issue that without #2588 the response isn't handled correctly?
So what should I do to get the correct response? I can't fix the Apache setup as it isn't my website.

asvetlov · 2018-02-01T06:47:45Z

Quoting @fafhrd91:

“upgrade: h2” is standard http/2 upgrade process for clear connection upgrade. Spec states that server should respond with http/1 if it does not understand http/2.

aiohttp client ignores HTTP/2 upgrade responses and continues to work on HTTP/1.1 protocol version.

fafhrd91 · 2018-02-01T06:56:56Z

I think this is the problem https://github.com/aio-libs/aiohttp/blob/master/aiohttp/http_parser.py#L327

if server tries to upgrade connection to http/2, it sends "connection: upgrade" and "upgrade: h2" headers. aiohttp in this case mark connection as upgraded and tries to use custom parser (i.e. websocket)
but according to spec it should not do this and just need to continue to use http/1.1

that is my understanding of this issue

lock · 2019-10-28T10:02:35Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new

asvetlov added this to the 3.0 milestone Oct 19, 2017

lphuberdeau mentioned this issue Dec 4, 2017

Limit protocol upgrades to supported protocols #2588

Closed

5 tasks

asvetlov mentioned this issue Jan 27, 2018

Request content is not returned on specific URL #2076

Closed

asvetlov added a commit that referenced this issue Jan 30, 2018

Add a note for #2277

c6e754b

asvetlov closed this as completed Jan 30, 2018

hellysmile mentioned this issue Feb 7, 2018

No content #2711

Closed

lock bot added the outdated label Oct 28, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available #2277

Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available #2277

segal90 commented Sep 18, 2017

asvetlov commented Sep 20, 2017

segal90 commented Sep 20, 2017

Findus23 commented Jan 27, 2018

asvetlov commented Jan 27, 2018

Findus23 commented Jan 27, 2018

asvetlov commented Jan 30, 2018 •

edited

Loading

Findus23 commented Jan 30, 2018

Findus23 commented Jan 30, 2018

asvetlov commented Jan 30, 2018

Findus23 commented Jan 30, 2018

asvetlov commented Feb 1, 2018

fafhrd91 commented Feb 1, 2018

lock bot commented Oct 28, 2019

Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available #2277

Missing response content if header "Upgrade: h2,h2c" and "Connection: Upgrade" are available #2277

Comments

segal90 commented Sep 18, 2017

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

asvetlov commented Sep 20, 2017

segal90 commented Sep 20, 2017

Findus23 commented Jan 27, 2018

asvetlov commented Jan 27, 2018

Findus23 commented Jan 27, 2018

asvetlov commented Jan 30, 2018 • edited Loading

Findus23 commented Jan 30, 2018

Findus23 commented Jan 30, 2018

asvetlov commented Jan 30, 2018

Findus23 commented Jan 30, 2018

asvetlov commented Feb 1, 2018

fafhrd91 commented Feb 1, 2018

lock bot commented Oct 28, 2019

asvetlov commented Jan 30, 2018 •

edited

Loading