aiohttp TLS websocket fails for continously sending packages on ESP32 #851
Comments
|
@wohltat your test does not do what you think is doing e.g. # allocate and prepare fixed block of data to be sent
b = bytearray(10_000)
for k in range(100):
p = k*100
b[p:p] = b'X'*96 + f'{k:3}' + '\n'
mv = memoryview(b)if you add print(len(b), b[-100:])You will see that, 1) it has a length of 20_000 and 2) there are invalid UTF bytes. Also here ...
await ws.send_str(b[0:100*n].decode())
n += 1
...this is sending an infinite growing string, so I'm not sure what you want to test with this... anyway here is a proper test (working on unix and esp32 ports) import aiohttp
import asyncio
import sys
import time
# allocate and prepare fixed block of data to be sent
b = bytearray(10_000)
for k in range(100):
p = k * 100
b[p : p + 100] = b"X" * 96 + f"{k:3}" + "\n"
mv = memoryview(b)
# print(len(b), b[-100:])
try:
size = int(sys.argv.pop(-1))
except Exception:
size = 2000
# sys.exit()
URL = "wss://<URL>:8448/echo"
sslctx = False
if URL.startswith("wss:"):
try:
import ssl
sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
sslctx.verify_mode = ssl.CERT_NONE
except Exception:
pass
async def ws_receive(ws: ClientWebSocketResponse):
try:
async for msg in ws:
if msg.type != aiohttp.WSMsgType.TEXT:
print("type:", msg.type, repr(msg.data))
except (TypeError, OSError) as e:
print("ws_receive", repr(e))
async def ws_test_echo(session):
n = 0
l = 0
m = 0
ws_receive_task = None
async with session.ws_connect(URL, ssl=sslctx) as ws:
ws_receive_task = asyncio.create_task(ws_receive(ws))
try:
while True:
t0 = time.ticks_ms()
while (n * 100) + size < len(b):
await ws.send_str(b[n * 100 : (n * 100) + size].decode())
dt = time.ticks_diff(time.ticks_ms(), t0) / 1e3
print(
"-------------------",
f"packet # {m} |",
f"total: {l/1000:.1f} KB |",
f"{(size/1000)/dt:.1f} KB/s |",
f"{((size*8)/dt)/1e3:.1f} kbps",
"------------------------",
end="\r",
)
n += 1
m += 1
l += size
t0 = time.ticks_ms()
await asyncio.sleep_ms(100)
n = 0
except KeyboardInterrupt:
pass
finally:
await ws.close()
async def main():
async with aiohttp.ClientSession() as session:
print(f"session test @ packet size: {size/1000:.1f} KB")
await ws_test_echo(session)
def run():
asyncio.run(main())
if __name__ == "__main__":
run()and server import ssl
from microdot import Microdot
from microdot.websocket import with_websocket
app = Microdot()
html = """<!DOCTYPE html>
<html>
<head>
<title>Microdot Example Page</title>
<meta charset="UTF-8">
</head>
<body>
<div>
<h1>Microdot Example Page</h1>
<p>Hello from Microdot!</p>
<p><a href="/shutdown">Click to shutdown the server</a></p>
</div>
</body>
</html>
"""
@app.route("/")
async def hello(request):
return html, 200, {"Content-Type": "text/html"}
@app.route("/echo")
@with_websocket
async def echo(request, ws):
t0 = time.ticks_ms()
l = 0
m = 0
while True:
data = await ws.receive()
size = len(data)
l += size
dt = time.ticks_diff(time.ticks_ms(), t0) / 1e3
print(
"-------------------",
f"packet # {m} |",
f"total: {l/1000:.1f} KB |",
f"{(size/1000)/dt:.1f} KB/s |",
f"{((size*8)/dt)/1e3:.1f} kbps",
"------------------------",
end="\r",
)
m += 1
t0 = time.ticks_ms()
# print(data[-4:].replace("\n", ""), end="\r")
await ws.send(data)
@app.route("/shutdown")
async def shutdown(request):
request.app.shutdown()
return "The server is shutting down..."
ssl_cert = "cert.pem"
ssl_key = "key.pem"
sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
sslctx.load_cert_chain(ssl_cert, ssl_key)
app.run(port=8448, debug=True, ssl=sslctx)client server |
|
Hi, thanks for quick your answer. Your are right with the size of the bytearray, but the first 10_000 bytes stay the same so i can't see how this influences the test.
This was my intention. I wanted to test with varying and increasing packet sizes. Your test has the same size of 2kb for every packet which was never a problem on the local server and is ocasionally failing on the remote server (that i don't know exactly what it is doing). With increasing packet sizes the point of failure is almost always at (or very close to) 4k for the local server. Strangely the connection to the remote server fails rather randomly between 2k and 8k. But i don't know what kind of server that is. The other problem that may be connected is that there is a memory problem, that may be caused by memery fragmentation i guess. I have no other explanation since there should be enough memory available. When i use a little memory in between the packets than the following memory error might occur. Not always though, often the connections breaks because of the before mentioned errors: I used the following client code on the ESP32 (The server ran on my pc under linux). The list import aiohttp
import asyncio
import gc
# allocate and prepare fixed block of data to be sent
b = bytearray(10_000)
for k in range(100):
p = k*100
b[p:p+100] = b'X'*96 + f'{k:3}' + '\n'
mv = memoryview(b)
print(len(b), b[-100:])
l = []
URL = "wss://somewebsocketserver"
sslctx = False
if URL.startswith("wss:"):
try:
import ssl
sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
sslctx.verify_mode = ssl.CERT_NONE
except Exception:
pass
async def ws_receive(ws : ClientWebSocketResponse):
try:
async for msg in ws:
if msg.type != aiohttp.WSMsgType.TEXT:
print('type:', msg.type, repr(msg.data))
except TypeError as e:
print('ws_receive', repr(e))
async def ws_test_echo(session):
n = 1
while True:
ws_receive_task = None
async with session.ws_connect(URL, ssl=sslctx) as ws:
ws_receive_task = asyncio.create_task(ws_receive(ws))
try:
while True:
gc.collect()
print(f'{gc.mem_free()=} {gc.mem_alloc()=}')
print('-------------------', n, '------------------------')
await ws.send_str(b[0:100*n].decode())
n += 1
l.append('abcdefghijkl' + str(n))
await asyncio.sleep_ms(300)
except KeyboardInterrupt:
pass
finally:
await ws.close()
async def main():
async with aiohttp.ClientSession() as session:
await ws_test_echo(session)
if __name__ == "__main__":
asyncio.run(main()) |
Because at some point it was sending non-UTF characters so it looked like data corruption.
Ok I see, I'm testing this in unix port at both sides, with increased sizes and at some point it "breaks" .i.e at 8000-10000 for |
|
I tried it again with a python-websockets server described in #853 (comment). The program also freezes and i get the following on the server side: or also this |
|
The websockets implementation is based on https://github.com/danni/uwebsockets ,so it may be interesting to test that and see if it breaks too. Using Wireshark may help 🤔, and something like iperf3 for websockets would be nice.... also there is micropython/micropython#12819, and https://github.com/orgs/micropython/discussions/14427#discussioncomment-9337471 [EDIT] this may help too So it looks like a wifi beacon timeout... |
|
I took a closer look in wireshark and found something i didn't expect. The TLS packets with increasing size are probably the ones with the payload. As one expect they increase by 100 byte every time.
When the connection breaks there are those TCP packets missing. This causes OSError 113 (= ECONNABORTED). As i understand it OSError is not from micropython but the Espressif IDF. Is this maybe some kind of connection-alive-checking mechanism that could be turned of? |
I've tried to increase the beacon timeout but it only takes longer to throw the error after the socket becomes unresponsive, it seems like the whole network stack stops...I've just tested the solution at micropython/micropython#12819 but it still stops at some point...same with enabling debug/verbose mode, it stops too...well I think this may be important to read it carefully... https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/wifi.html#wi-fi-buffer-usage |
|
@wohltat my conclusion is that after micropython/micropython#13219 and micropython/micropython#12141 (see e.g. micropython/micropython#12141 (comment))
see also possible solutions:
|
|
I've spend some time researching. So far i don't have a complete solution but some insights:
|
|
I also tried compiling with IDF v5.2, which is supported according to the micropython documentation.
|
Yes the problem is basically this:
Solution is preallocate what you really need in MicroPython and make sure it does not allocate over a certain size leaving enough memory for idf heap/wifi buffers. Also be aware that slicing e.g.
yes but it will depend on the number of core/threads of your computer use
IDF v5.1.2 it's the latest I've tested which seems to work. |
@Carglglz
I have an application i want to send TLS / SSL packages over websockets using aiohttp on an ESP32. The problem is that the websockets fail after a short while when using packages that have a little bigger size around 2kB to 4kB.
Here is a simple test script:
remote
I'll get the following exception after a while. Just dealing with the exception is not satisfying since it takes a while and the applicaiton is blocked in that time.
(OSError: 113 = ECONNABORTED)
Noteworthy is that there are always some packages that micropyhon thinks are sent already but don't reach the other side.
I also do not receive a websocket close package.
This was tested on a remote Websocket server that i don't control.
local
When i try it on a local server, i see different problems.
(OSError: 104 = ECONNRESET)
The servers seems to not receive the complete message / only a corrupted message and closes the connection:
The closing of the websocket is also not handled properly by the micropython application / aiohttp. The close package is received but the connection is not closed automatically.
Another problem is that even if i deal with the exceptions and reconnect automatically, my application will eventually crash because of a run out of memory.
This is really problematic to me since i only send messages of size 2k or so.
I guess this is because of the not so memory economical design of aiohttp. For sending and receiving there is always a lot of new memory allocation involved.
The use of preallocation and
memoryviewseems reasonable here.This is the local python websocketserver code:
MicroPython v1.23.0-preview.344.gb1ac266bb.dirty on 2024-04-29; Generic ESP32 module with ESP32
The text was updated successfully, but these errors were encountered: