Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
As it turns out, aiohttp does not handle json streams well.
Initially, we used
iter_any
to iterate through the logs, but the method implies\r\n
delimiter (and RLP gateway messages are separated by\n\n
), so some messages were skipped or merged into one.The current solution is iterating through messages separated by
\n
. That should work fine for json streams, however there is a size limitation for messages - 128kb. So, yet again, a few messages are being skipped. The issue is know - rtfol/aiohttp-sse-client#11, aio-libs/aiohttp#4453,Of course, I would prefer aiohttp to extend the functionality - either add a
separator
argument or a flexible size limit. However, it can take time.In the meantime, I wrote a workaround, which allows to iterate through small chunks of data and separate messages by
\n\n
. Let me know what you think.Thanks in advance!