optimal number of ClientSessions?

[jreadey]

I have a service where a container is sending http PUT requests to 4 to 100 other containers. The payload is 2-4 MB of binary data per request.

When the container is under heavy load I'm getting occasional ClientErrors like "Can not write request body", or "Connection reset by peer". Requests generally succeed with some retry logic to handle these errors, but would like to avoid if possible.

I tried various approaches to setting up the ClientSession to try to resolve this: 1 shared session, 1 session per url, pool of sessions, etc., but always get these errors. The only thing that seems to help is to create a ClientSession per request, but that slows things down (even more than the retry logic).

Is there some approach to eliminate (or largely eliminate) the ClientErrors without effecting performance?

Async logic for the service is here: https://github.com/HDFGroup/hsds/blob/master/hsds/chunk_crawl.py#L556 if that helps

[Dreamsorcerer]

They sound like server-side errors to me, perhaps overloading the servers?

But, a couple of ideas, maybe look at using aiojobs to schedule tasks and limit the number of concurrent tasks being run.

Also, check that you are not doing anything CPU intensive that might block your application and cause timeouts (Python should warn you if you enabled dev features e.g. with python -Wall -X dev).