Issue with ClientSession.request('POST') method

[ja8zyjits]

Long story short

Iam trying to build a forwarder with aiohttp but iam unable to forward POST request.

Expected behaviour

if I send a post request to http://httpbin/post i normally get the full content.

Actual behaviour

But when I use the forwarder to send request through the forwarder it waits a few seconds till timeout and it provides ConnectionError: ('Connection aborted.', BadStatusLine("''",)), later if I increase the timeout it provides a huge error

Error handling request
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 662, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/client.py", line 206, in _request
    yield from resp.start(conn, read_until_eof)
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 598, in start
    message = yield from httpstream.read()
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/streams.py", line 640, in read
    result = yield from super().read()
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/streams.py", line 475, in read
    yield from self._waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
aiohttp.errors.ServerDisconnectedError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/server.py", line 265, in start
    yield from self.handle_request(message, payload)
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/web.py", line 96, in handle_request
    resp = yield from handler(request)
  File "asyncapp.py", line 14, in handle
    async with session.request(request.method, url, headers=request.headers) as response:
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/client.py", line 558, in __aenter__
    self._resp = yield from self._coro
  File "/home/jitesh/projects/proxy_ui3/lib/python3.5/site-packages/aiohttp/client.py", line 213, in _request
    raise aiohttp.ClientResponseError() from exc
aiohttp.errors.ClientResponseError

• If I send post request to a dummy server running in my local host(with flask) iam able to perform a post request
• I thought it might be a problem with the server to whom iam sending, but after tryng pastebin, httpbin all failed.

Steps to reproduce

I have hereby added my server code

from aiohttp import web
import aiohttp
import asyncio
import async_timeout


loop = asyncio.get_event_loop()

async def handle(request):
    url = request.rel_url
    print(request.method)
    async with aiohttp.ClientSession(loop=loop) as session:
        with async_timeout.timeout(10):
            async with session.request(request.method, url, headers=request.headers) as response:
                print(response.headers)
                html = await response.read()
                headers = dict(response.headers)
                status = response.status
            if 'Content-Encoding' in headers:
                del headers['Content-Encoding']
        return web.Response(body=html, headers=headers, status=status)

app = web.Application(loop=loop)
app.router.add_route('GET', '/{url:.*}', handle)
app.router.add_route('POST', '/{url:.*}', handle)
web.run_app(app, port=5000)

• Run this server and it the next terminal use requests to send a request

>>> import requests
>>> p =  {'http': 'http://127.0.0.1:5000', 'https': 'http://127.0.0.1:5000'}
>>> response = requests.get("http://httpbin.org/ip", proxies=p) # works
>>> response = requests.post('http://httpbin.org/post', json={"key": "value"}, proxies=p)  # fails
>>> response = requests.post('http://127.0.0.1:8000', json={"key": "value"}, proxies=p) # request to my local falsk server, it works

Your environment

Its ubuntu 16.04 with a python3.5 virtual env

[popravich]

There is seem to be an infinite loop — your handler takes all request headers including host and makes
client request to the same host and url, basically to itself.

[ja8zyjits]

well then how does post work with the normal localhost that i have here running?
and also get request is being successful but not post request alone.

[popravich]

Sorry, my mistake.
Anyway your code works for me

[ja8zyjits]

You mean the code is properly sending the POST request? and it is not just working for me?
Does it has something to do with my environment or any thing to do with libraries and versions?
I have hereby added my pip freeze

aiohttp==1.1.2
aiohttp-utils==2.0.1
astroid==1.4.8
async-timeout==1.1.0
chardet==2.3.0
click==6.6
decorator==4.0.10
Flask==0.11.1
gunicorn==19.6.0
ipython==5.1.0
ipython-genutils==0.1.0
isort==4.2.5
itsdangerous==0.24
Jinja2==2.8
lazy-object-proxy==1.2.2
MarkupSafe==0.23
mccabe==0.5.2
multidict==2.1.2
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.9
ptyprocess==0.5.1
Pygments==2.1.3
pylint==1.6.4
python-mimeparse==1.6.0
redis==2.10.5
redis-loghandler==0.3
requests==2.11.1
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.1
wcwidth==0.1.7
Werkzeug==0.11.11
wrapt==1.10.8
yarl==0.7.0

[popravich]

Heh, re-read again and wound issue)
Your handler does not check that request has any body and doesn't send any.
You need to check request.has_body and add data parameter to client request if body is present.

[popravich]

this should work:

async def handle(request):
    url = request.rel_url
    print(request.method)

    kw = {"headers": request.headers}
    if request.has_body:
        kw["data"] = request.read()

    async with aiohttp.ClientSession(loop=loop) as session:
        with async_timeout.timeout(10):
            async with session.request(request.method, url, **kw ...

[ja8zyjits]

That worked, thank you!!
One small question more, then why did it work with my small testing server. Is data parameter that important?
I know data parameter is very important but in normal libraries like python requests it doesn't raise any kind of error even if we dont add any body part in the POST request.
I forward the request from the aiohttp server code mentioned above, to this small server without any body part, it worked.

>>> response = requests.post('http://127.0.0.1:8000', json={"key": "value"}, proxies=p) # request to my local falsk server, it works

My test server, it was working without adding the data parameter.

from flask import Flask

app = Flask(__name__)

@app.route("/", methods=['GET', 'POST'])
def home():
    return "True"

if __name__ == "__main__":
    app.run(port=8000, debug=True)

That brings me to a suggestion, isn't it ok for a client to send a POST request without a body, shouldn't we have provisions for checking the data part implicitly? and ignoring it rather than showing an error, that we(as of people like me) hardly understand?

[asvetlov]

@ja8zyjits please correct me if I misunderstood you.

You are suggesting to raise an exception early if request type is POST but no data (request's BODY) is provided.
Is it correct?

[ja8zyjits]

@asvetlov
1] in general python requests doesn't raise any error or exception in POST if a data parameter is missing.
Why do we have a problem here?
2] in my example test server is accepting POST request from my aiohttp client without any data parameter, why does it work without any error?

[popravich]

2] …why does it work without any error?

Your local flask app works because it does not read POST data (flask, actually werkzeug, does it in lazy manner)

[asvetlov]

I believe the question could be closed.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.