Optimal chunk size for streaming response content

[fgallaire]

What is the optimal chunk size for streaming response content?

It's not indicated in the example from the official documentation:

with open(filename, 'wb') as fd:
    async for chunk in resp.content.iter_chunked(chunk_size):
        fd.write(chunk)

I'm currently using the chunk size of 1024.
Should I use a larger one? Why?

[Dreamsorcerer]

I could be wrong, but I've always assumed the tradeoff is between performance and memory use. I have a backup program which needs to sync large amounts of data in one go (100MBs or GBs), for this I set the chunksize to 2MB (the library I was using defaulted to 1 byte and was incapable of exceeding 10KB/s due to that). For an embedded device with very limited memory, I imagine a chunksize of 1KB or less would be more appropriate. For the average application, I'm guessing around 1-10KB is safe.

However, that's mostly my intuition, maybe somebody else with more knowledge can add some information.

[asvetlov]

I concur with @Dreamsorcerer

Read buffer sizes for TCP Linux socket are: 4KB min, 85KB default, 4MB max. Linux dynamically adjusts the size.

Values lesser than 4KB don't add much value from the performance point of view.
4-8 (or even 16) MB can keep the read socket buffer empty and ready for new data packets.

The exact best number depends on your system and requires testing.

[fgallaire]

Thanks @Dreamsorcerer and @asvetlov
Could we say that 4KB is better for low memory device, and 4MB faster ?
Which value do you use yourself ?

[fgallaire]

@asvetlov watching no performance improvement with larger chunks but found a bug:
resp.content.iter_chunked(chunk_size) doesn't work fine for chunk_size > 16*1024

One example of the real chunk sizes for chunk_size = 85*1024:

16384
32768
16384
16384
49152
32768
87040
76800
87040
87040
87040
1024
87040
87040
87040
1024

[fgallaire]

Same problem with read() #3681

[fgallaire]

Seems to be the normal way for aiohttp... what a crap!

[fgallaire]

Was OK up to 16*1024 because it was the chunk size of my test server.
Just moved to iter_chunks().

[asvetlov]

Yes, the chunk size is sent by a peer.
If it is greater than chunk_size argument -- the chink is split to avoid uncontrollable memory usage.