Issue: "Too many open files" error despite using TCPConnector with limit

[loic-bellinger]

Hello,

I am encountering the classic system error "Too many open files" / "too many file descriptors" when making a large number of requests (50K+) to different domains. Despite setting up a TCPConnector with a limit=500, the error is still triggered. For reference, ulimit -n on my system returns 1024.

I suspect that the sockets aren't being closed properly, which causes the file descriptor limit to be reached. Using force_close=True on the TCPConnector seems to solve the problem, but I assume this introduces a lot of overhead by not reusing connections.

Given my session configuration (provided below), do you see a solution to ensure that I stay within the file descriptor limits of my system? I've also tried using asyncio.BoundSemaphore, but it didn’t seem to make a difference. In any case, I assume it's redundant with the limit parameter on the TCPConnector, correct?

Here’s how I am configuring my session:

resolver = AsyncResolver(nameservers=[...])

connector = aiohttp.TCPConnector(
    limit=500, 
    limit_per_host=4,
    ssl=False,
    resolver=resolver,
    ttl_dns_cache=300
)

# See https://github.com/aio-libs/aiohttp/issues/3203#issuecomment-544136905 for the reason why total is set to None
timeout = aiohttp.ClientTimeout(
    total=None, connect=None, sock_connect=40, sock_read=40, ceil_threshold=5
)

async with aiohttp.ClientSession(
    raise_for_status=True,
    connector=connector,
    timeout=timeout,
    cookie_jar=aiohttp.DummyCookieJar(),
) as client:
    # request logic here

Do you have any advice on how to properly handle the file descriptor limits without forcing the closure of every socket?

Thanks in advance!

[Dreamsorcerer]

It would be good if you could try this on master. There's a possibility it's fixed there and couldn't be backported to 3.x releases.