aiohttp - concurrent requests are getting hung #3698
Comments
|
First, import aiohttp
import asyncio
async def fetch(session):
async with session.get("https://httpstat.us/200") as response:
print(await response.text())
async def main(n):
async with aiohttp.ClientSession() as session:
await asyncio.gather(*(fetch(session) for _ in range(n)))
asyncio.run(main(100))I've checked exactly your example and everything is the same: no hangs. Please check your firewall/gateway |
|
Feel free to reopen issue. I'm closing now since can not be reproduced. |
@socketpair I've ruled out the possibility of firewall/gateway issues. Moreover, I don't see any hangs if I create a new With this code, I never see concurrent requests. That is, after 20-30 requests, the program just hangs. Following works like charm. No hangs!! I am also seeing a frequent SSL errors which I believe is a known bug with aiohttp 3.5.4 on Python 3.7.x. However, I don't think the code hangs because of the SSL errors. When I use a persistent session to make concurrent requests, it sometimes hangs and completes after a very long time and sometimes it fails with following traceback. I am not sure why |
|
@socketpair looks like this SSL error is a bug that affects aiohttp/asyncio on python3.7. No fix yet. More info: #3535 |
This comment has been minimized.
This comment has been minimized.
|
I have the same problem, even when opening a new
If needed, I can provide more information. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Same issue. Using one persistent session (ssl=False) for entire script: aiohttp makes 10 concurrent connections, then hangs, then makes random quantity of concurrent requests. Notice the timing: Image
Image
Script finishes in less than a second (4,60s vs 0.60s). |
|
The main problem with creating a single In my case (Python 3.8.2), a single GET request to https://diego.assencio.com already caused into this: and the result was "requests succeeding but slowly", compared to "not a single request succeeding". Interestingly, the idea from @raees-khan produced the exact same outcome for me: requests started succeeding, but just as slowly as with a single instance of For the record, HTTP requests are working fine. All issues I observed happen with HTTPS only. |
|
I'm also hitting this with my matrix bot, it will work for a long time and then crash with: Seperate clientSession per request: async def _lab(self):
try:
topic = self.args[0]
if topic == "light":
msg = await self._lab_light()
await send_text_to_room(self.client, self.room.room_id, msg)
elif topic == "wifi":
msg = await self._lab_wifi()
await send_text_to_room(self.client, self.room.room_id, msg)
else:
first_msg = "You did not specify which information you want, so I'm gonna guess it as both."
await send_text_to_room(self.client, self.room.room_id, first_msg)
light_msg = await self._lab_light()
await send_text_to_room(self.client, self.room.room_id, light_msg)
wifi_msg = await self._lab_wifi()
await send_text_to_room(self.client, self.room.room_id, wifi_msg)
except IndexError:
pass
async def _lab_light(self):
async with aiohttp.ClientSession() as session:
async with session.get("http://localhost/pi_api/gpio/?a=readPin&pin=1") as response:
api_response = await response.json()
api_data = api_response['data']
if api_data == 0:
msg = "Someone is probably at the lab as the light is on."
else:
msg = "Nobody is at the lab, the light is currently off."
return msg
async def _lab_wifi(self):
async with aiohttp.ClientSession() as session:
async with session.get("http://localhost/visitors/api/v1/visitors.php?format=json") as response:
api_response = await response.json()
if not api_response:
msg = "Nobody is using the Wifi that I know of currently."
else:
msg = "Someone or multiple someones are using the Wifi at the lab currently."
return msg |
|
I can't see any relation between the code you've posted and that traceback. The traceback suggests that the timeout occurred within the nio library. I'd assume if you are dealing with a federated service, then timeouts could be pretty common, so not sure how that library is supposed to deal with this. It appears there is a config option to change the timeout parameter though: https://github.com/poljar/matrix-nio/blob/master/nio/client/async_client.py#L300 |
|
I have the same issue. However, if I run the same script on the local windows 10 machine, it is working fine. |
|
I've got the same issue on Ubuntu 18.04. It's not safe to use ClientSession from several coroutines asyncroneously ( |
|
So I've been having these issues for ages, and it seems, at least in my exact use-case, to only affect WSL. I've tried to trace the cause of the issue as far as I can, but I can't seem to work out if it goes any further. This is for 3.8.1 -- I would try with 4.0.0a but it physically will not install no matter what I do. This is from a https://github.com/aio-libs/aiohttp/blob/v3.8.1/aiohttp/client.py#L353 That's as far as I can trace before VSCode and GitHub can only find stuff in pyi or test files respectively. If anyone can do a better job at tracing that, let me know. I'm kinda hoping my time doing this is wasted and it'll be fixed in v4.0.0a anyways, but just leaving this here for further investigation. |
|
This is also happening for me in 2022 |
|
Me too, the same on Ubuntu WSL (desktop) & Centos 7 (aws) when using aiohttp to make web server, even Tried another client |
|
Found out my case, it's browser's 6 pending reqs / server limit, not bcoz of aiohttp |
|
The original reproducer doesn't seem to have a problem when I run it. If this is still an issue, can anyone provide a reproducer (or test in CI if it affects a specific environment)? |
Dreamsorcerer
added
the
needs-info
|
Experiencing this exact same issue, but strangely, it only happens when I am connected to the Wireguard VPN network I have set up. I ran some tests to confirm and this behaviour only happens with the from datetime import datetime
import asyncio
import aiohttp
url = "https://official-joke-api.appspot.com/jokes/programming/random"
async def request(session, i=0):
print(str(i).zfill(2), "request start ", datetime.now())
async with session.get(url) as response:
data = await response.text()
print(str(i).zfill(2), "request return", datetime.now())
return data
async def main():
async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=100)) as session:
payloads = await asyncio.gather(*[request(session, i=i) for i in range(15)])
return payloads
results = asyncio.run(main())
for result in results:
print(result)When I run the above script whilst Wireguard is up, I notice that half of the requests return responses immediately. However, every other request always hangs for up to 2-3 minutes. Even more strangely, it always seems to be every 2nd request which hangs. Example output showing this behaviour where every odd numbered request hangs is below. As with raees-khan, if I introduce a To eliminate Wireguard as being the issue, I ran the following bash script for concurrent requests and found they executed concurrently without issue. #!/bin/bash
send_request() {
echo $i 'request start ' $(date)
curl -s 'https://official-joke-api.appspot.com/jokes/programming/random' 1>/dev/null
echo $i 'request return' $(date)
}
for i in $(seq 1 15); do
send_request &
doneFor context if it helps, here's the Wireguard config on the machine I am experiencing the issue on and the connected upstream server's config. [Interface]
PrivateKey = ***
Address = 10.2.0.12/32
DNS = 10.2.0.2
MTU = 1420
# ping public IP
PreUp = ip route add $(curl -s ifconfig.me) via $(ip route | grep default | awk '{print $3}') dev $(ip route show default | awk '{ print $5; exit }')
PostDown = ip route del $(curl -s ifconfig.me) via $(ip route | grep default | awk '{print $3}') dev $(ip route show default | awk '{ print $5; exit }')
# route public SSH traffic around VPN
PreUp = ip rule add table 128 from $(ip addr show $(ip route | awk '/default/ { print $5 }') | grep "inet" | head -n 1 | awk '/inet/ {print $2}' | cut -d'/' -f1)
PreUp = ip route add table 128 to $(ip -o -f inet addr show | awk '/scope global/ {print $4}' | head -n1 | sed -r 's:([0-9]\.)[0-9]{1,3}/:\10/:g') dev $(ip route show default | awk '{ print $5; exit }')
PreUp = ip route add table 128 default via $(ip route | grep default | awk '{print $3}')
PostDown = ip route del table 128 default via $(ip route | grep default | awk '{print $3}')
PostDown = ip route del table 128 to $(ip -o -f inet addr show | awk '/scope global/ {print $4}' | head -n1 | sed -r 's:([0-9]\.)[0-9]{1,3}/:\10/:g') dev $(ip route show default | awk '{ print $5; exit }')
PostDown = ip rule del table 128 from $(ip addr show $(ip route | awk '/default/ { print $5 }') | grep "inet" | head -n 1 | awk '/inet/ {print $2}' | cut -d'/' -f1)
[Peer]
PublicKey = ***
PresharedKey = ***
Endpoint = ***
AllowedIPs = 0.0.0.0/1, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/6, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/9, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/6, 200.0.0.0/5, 208.0.0.0/4, 224.0.0.0/3, ::/0
PersistentKeepAlive = 25Upstream server config: [Interface]
PrivateKey = ***
Address = 10.2.0.2/24
DNS = 127.0.0.1
ListenPort = ***
MTU = 1400
PreUp = ip route add $(curl -s ifconfig.me) via $(ip route | grep default | awk '{print $3}') dev $(ip route show default | awk '{ print $5; exit }')
PostDown = ip route del $(curl -s ifconfig.me) via $(ip route | grep default | awk '{print $3}') dev $(ip route show default | awk '{ print $5; exit }')
PostUp = ufw allow Wireguard comment 'Managed by Wireguard'
PostUp = ufw allow from *** to any app SSH comment 'Managed by Wireguard'
PostUp = ufw allow from *** to any app DNS comment 'Managed by Wireguard'
PostUp = ufw route allow in on %i out on %i to any comment 'Managed by WireGuard'
PostDown = ufw delete allow Wireguard
PostDown = ufw delete allow from *** to any app SSH
PostDown = ufw delete allow from *** to any app DNS
PostDown = ufw route delete allow in on %i out on %i to any
PostUp = ip rule add from *** lookup main
PostUp = iptables -w -t nat -A POSTROUTING -o %i -j MASQUERADE
PostUp = ip6tables -w -t nat -A POSTROUTING -o %i -j MASQUERADE
PostDown = ip rule delete from *** lookup main
PostDown = iptables -w -t nat -D POSTROUTING -o %i -j MASQUERADE
PostDown = ip6tables -w -t nat -D POSTROUTING -o %i -j MASQUERADE
[Peer]
PublicKey = ***
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = ***
[Peer]
PublicKey = ***
PresharedKey = ***
AllowedIPs = 10.2.0.12/32Given the results of the bash script, I believe the cause of the issue lies with |
|
OK, first, curl is a terrible comparison as it's not using a connection pool or anything. If you move the ClientSession into request(), so you have a new session for every request, then you'll have a fairer comparison with curl. I suspect the problem disappears when you do that. So, if you can confirm that, we can ignore curl going forward. If you can also check with a single ClientSession, but with Then, with the original version, I'd like to try and see which connections are being used, so try and add a print statement in: Note that when the response is small, you'll get a None because the connection is already released. Ideally, try and reproduce on a URL that returns larger responses so you don't have any Nones. I can only imagine that something has gone wrong in the wg connection or the server, causing it to not send any response until a timeout occurs or something. I can't imagine that aiohttp is failing to send the request or respond to the data received for a long time (we'd throw a TimeoutError or similar exception if that were the case). I'll also try wg tomorrow and see if I can reproduce anything. Just to confirm, you're using 3.10.5? |
|
Thanks for the quick response. However, I have to confess I checked again and I was using 3.9.5 previously. I upgraded to 3.10.5 and ran the test again and found the issue had vanished. I can't say exactly what the issue was, but I'm glad it appears to have been resolved already! For posterity, I've added my analysis using 3.9.5 from your suggestions below anyhow in case it may still be of use. Thank you again for your help!
|
|
Yeah, we fixed a number of things in recent releases, so it may have been caused by something in the parser messing up the HTTP messages in subtle ways on rare cases. So, I think we'll consider this fixed then. |
|
@geo-martino if you have time it would be interesting to find which release fixed it by doing a binary search over the releases |
|
If using the Python parser, then #8722 would be a likely candidate. Could result in the connection appearing to hang. |
|
@thehesiod Just checked using the same script again and it appears to have been fixed in 3.10.0 as I don't see the same issue in any version >= 3.10.0 |
Concurrent requests are getting hung. Here's a sample code that I am using to test concurrent requests.
When I make 10 concurrent requests, the first 4-5 requests are made concurrently and then it gets hung for over 10 seconds and then starts the remaining tasks which get hung again after running 2-3 concurrent requests. If I make 100 concurrent requests, it makes about 25-30 concurrent requests and gets hung and then makes 5-6 requests and gets hung again, it does this until all tasks are complete.
It's taking over two minutes to make 100 requests to https://httpstat.us/200 with aiohttp.
If I don't use a persistent ClientSession and create new ClientSession for every request, then all hundred requests finish within 5 seconds without getting hung.
I am not sure what I am doing here. Any help will be highly appreciated.
I am running Python 3.7.2 and aiohttp 3.5.4
The text was updated successfully, but these errors were encountered: